Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/10—Segmentation; Edge detection
  • G06T7/11—Region-based segmentation
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/10—Segmentation; Edge detection
  • G06T7/143—Segmentation; Edge detection involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/10—Segmentation; Edge detection
  • G06T7/155—Segmentation; Edge detection involving morphological operators
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/20—Image preprocessing
  • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
  • G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V20/00—Scenes; Scene-specific elements
  • G06V20/60—Type of objects
  • G06V20/69—Microscopic objects, e.g. biological cells or cellular parts
  • G06V20/695—Preprocessing, e.g. image segmentation
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10056—Microscopic image
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10064—Fluorescence image
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20036—Morphological image processing
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20076—Probabilistic image processing
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20112—Image segmentation details
  • G06T2207/20152—Watershed segmentation
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30004—Biomedical image processing
  • G06T2207/30024—Cell structures in vitro; Tissue sections in vitro

Definitions

• the invention relates generally to digital images and more specifically to segmentation of objects in the digital images to extract content from the images.
• Segmenting images of complex, three-dimensional materials into discrete and identifiable objects or targets for analysis is a challenging problem because of the high degree of variability associated with the materials, and inconsistencies between, and anomalies introduced by, the imaging systems themselves.
• Cells are three-dimensional objects, and the images of such cells capture a two-dimensional projection that corresponds to the given slice of the tissue. Partial cell volumes that are outside the focal plane are commonly observed. Nuclei shape and size also vary widely across different tissue types and even within the same tissue type. For example, the shape of epithelial cell nuclei in lung tissue is different than the shape of stromal cell nuclei in lung tissue. The grade of a given cancer also may significantly affect the shape and the size of the nuclei. For example, the size of the cell nuclei in breast cancer is a diagnostic indicator.
• staining quality and tissue processing also vary from sample to sample; although non-specific binding and tissue autofluorescence can be reduced, they typically cannot be eliminated; the image acquisition system further introduces noise, particularly, for example, if the image acquisition camera is not actively cooled; and most microscopes are manufactured with tolerances up to 20% non-uniformity of illumination.
• the methods of the invention provide a highly robust boosted approach wherein the technical effect is to segment images into discreet or targeted objects.
• the methods build a strong or reliably consistent segmentation result from a plurality of generally weaker or less consistent segmentation results.
• Each weaker segmentation method generates a probability map that captures different, yet complementary, information.
• the strong segmentation integrates the probability results from the weaker segmentation methods, based on various parameters or predefined rules such as, for example, a weighted average or sum.
• a watershed method is applied, together with one or more morphological constraints to the integrated, but stronger, combined segmentation, to identify and segment the nuclear regions of the image.
• the methods are first described here using a more general workflow, where weak segmentation algorithms are combined to generate a strong segmentation algorithm, that may be applied to a variety of images for a variety of purposes.
• the general method is then applied to a specific, but non- limiting, example in which an image of a biological sample is segmented into cells.
• segmentation algorithms comprising, curvature based segmentation, image gradients, Gabor filters, and intensity, that are particularly useful with images of biological materials
• the methods of the invention may be applied to other types of subject matter and so may comprise alternative subsets of algorithms.
• An embodiment of the method of the invention for segmenting a digital image into a plurality of target objects, comprises: generating a plurality of probability maps of the image, wherein each probability map is derived from a different segmentation classifier;
• FIG. 1 is a flow diagram of an example of the method and system of the invention for segmenting an image
• FIG. 2 is a flow diagram of a specific example of the method and system of the invention for segmenting an image
• FIGs. 3A-3D are examples of probability maps based four weak classifiers.
• FIG. 3A-3D are examples of probability maps based four weak classifiers.
• FIG. 3A was generated using a curvature-based classifier
• FIG. 3B was generated using a Gabor filter bank
• FIG. 3C was generated using a gradient classifier
• FIG. 3D was generated using an intensity classifier.
• FIG. 4A is an example of a probability map generated using an example of a strong segmentation classifier
• FIG. 4B is an example of a map showing the detected object centers
• FIG. 4C is an example of a weighted image using morphological constraints
• FIG. 4D is an example of a segmented image
• FIG. 4E is an example of a final segmented image merged with the mapped nuclei.
• FIG. 5A is an example of an original, unsegmented image with a portion outlined in a red square
• FIG. 5B is an example of a final segmented image generated using an example of a method of the invention with the same corresponding portion outlined in a red
• FIG. 5C is a magnified view of the outlined portion of FIG. 5 A
• FIG. 5D is a magnified view of the corresponding outlined portion of FIG. 5B.
• FIG. 6A is another example of an original, unsegmented image of a xenograft model with a portion outlined in a red square
• FIG. 6B is an example of a final segmented image generated using an example of a method of the invention with the same corresponding portion outlined in a red
• FIG. 6C is a magnified view of the outlined portion of FIG. 5A
• FIG. 6D is a magnified view of the corresponding outlined portion of FIG. 5B.
• the methods and systems provide significant improvements to previous methods for segmenting digital images.
• the methods in part construct a strong classifier from a number (N) of weak classifiers.
• the term weak classifier is used in this description merely to denote a classifier that, when used alone, does not provide a reproducibly strong, consistent segmented image, as does the stronger classifier of the invention which comprises a combination of a plurality of individual weaker classifiers.
• Each of the weaker classifiers, used in one or more of the embodiments described provides unique and different information in the form of a probability estimate whether a given pixel belongs to a target object, such as a nucleus of a cell.
• a combination classifier of the invention combines the results of the weaker individual classifier results.
• the stronger, classifier integrates both global and local information derived from the weaker segmentations to generate a more consistently accurate segmented image.
• a watershed algorithm is applied to the stronger data to identify and map individual target objects, such as cell nuclei.
• target object refers to any item of interest, to which a plurality of different classifiers or definitions can be applied, for the purpose of extracting content from a segmented digital image.
• classifier refers to one or more parameters of a digital image that can be expressed as an algorithm.
• the term "probability map” refers to a map, of all or a portion of the pixels or image points in a digital image, which indicates the likelihood that a given pixel falls within a class based on a classifier previously applied to the digital image.
• the map may be virtual, actual, stored or ephemeral, depending on a given application or system.
• the term "local object maxima” refers to the highest value or degree as defined by a given classifier, among the pixels or image points associated with a discrete target object in a digital image.
• object constraint refers to one or more algorithmic statements or rules that may be applied to an object that may include, but are not limited to, those that define or limit the object's context or situation; a property, attribute or characteristic of the object; and conditions or expression qualifiers.
• local object information refers to any information associated with a given object including, but not necessarily limited to, facts, data, conclusions, estimates, statistics, transformations, and conditions associated with an object.
• regional threshold refers to a rule or statement that is applied to an image to segment the image into regions such as, for example, watershed or watershed-based algorithms.
• digital device refers to any device that can at least temporarily store, display, generate, manipulate, modify or print a digital image.
• the methods and system may be used to segment of a broad class of objects in digital images.
• the methods and systems may be used, for example, to segment objects that have elliptical shapes, such as those found in images associated with industrial inspection and medical and biological imaging.
• One or more embodiments of the methods construct a probability map using a novel boosting approach, to which a watershed algorithm, with at least one object constraint, is applied.
• a strong classifier is constructed, together with one or more morphological constraints, based on a plurality of weaker classifiers that provide complementary information such as, but not limited to, shape, intensity and texture information.
• detection of cell nuclei comprises modeling various nuclei attributes such as shape, intensity and texture in the cell.
• nuclei texture when imaging cells, nuclei texture may vary, in part, due to uneven binding and distribution of the fluorescent dyes applied to the cellular material or tissue sample. Image intensity also varies between images and across a single and may be caused by a number of factors, some of which are associated with microscopy system itself.
• detection of two-dimensional cell nuclei obtained from three- dimensional tissue sections comprises modeling various nuclei attributes such as shape, intensity and texture in the cell.
• nuclei attributes such as shape, intensity and texture in the cell.
• shape, intensity and texture may vary, in part, due to uneven binding and distribution of the fluorescent dyes applied to the cellular material or tissue sample.
• Image intensity also varies between images and across a single and may be caused by a number of factors, some of which are associated with microscopy system itself.
• FIG. 1 An embodiment of the method of the invention, that is readily applicable to a variety of modalities and purposes, is generally shown and referred to in the flow diagram of FIG. 1 as method 10.
• three segmentation classifiers 12A, 12B, and 12C are applied in step 14 to image 16.
• classifier may comprise shape, intensity, and textural primitives.
• the resulting probability maps PI . PN generated by the three weaker segmentation classifiers 12A-12N are used to generate a stronger segmentation classifier 18 that is based on a weighted combination of the weaker classifiers.
• the method is not limited to using a specific number of weaker classifiers (e.g. Si . . . SN) or a specific number of weaker classifiers
• morphological constraints e.g. Mi . . .MN
• Individual, weaker classifiers may include, but are not limited to, shape features (such as, regular and irregular elliptical, circular, semi-circular shapes), intensity features (such as, homogeneity, histogram based-methods), textural features, (such as fractals, wavelets, second or higher order statistcs).
• Combination classifier 18 together with combined morphological constraints 20A, 20B and 20C, is then applied to image 16 using one or more regional thresholding algorithms, in step 22, together with one or more local constraints, to generate a resulting image 24, segmented at least in part into the target objects (e.g. cell nuclei).
• target objects e.g. cell nuclei
• the first step comprises generating a strong segmentation classifier using a plurality of weaker segmentation algorithms.
• the strong classifier is generated using a curvature based segmentation algorithm, two different gradient based segmentation algorithms and an intensity based algorithm, which can be expressed as follows:
• p CBS represents the probability map that is computed using a curvature based segmentation algorithm
• p Gabor represents the probability map that is computed using a Gabor filter based segmentation algorithm
• p Gradient represents the probability map that is computed using a gradient segmentation
• p f represents the probability map that is computed using an intensity based segmentation.
• the resulting probability map is a weighted average of the individual probability maps that are generated by the individual weak segmentation algorithms. The weights may be determined empirically or using supervised classification algorithms.
• the method comprises generating a probability map for each of the four weaker segmentation algorithms.
• the probability map (FIG. 3A), in this example, that is generated using a curvature based segmentation algorithm 34A represented by p CBS .
• the eigenvalues (x, ⁇ ), ⁇ 2 ( ⁇ , y) are numerically computed from: d 2 I(x, y) , d 2 I(x, y) r d 2 I(x, y) d 2 I(x, y) d 2 I(x, y)
• a probability map p CBS is estimated iteratively, where the probability that a pixel will belong in a blob-like structure is at a maximum when the pixel is at the center of the blob structure. Then a binary mask is estimated by selecting a threshold value, estimating the distance transform of the binary mask, where the response of the distance transform is at a maximum in the center of the bloblike structures and decreases outward toward the border or membrane of the cell or nucleus in this example.
• the image captures geometrical information derived from the defined filter bank and is suitable to detecting elliptical structures at different orientations.
• the image I Gabor captures the geometrical information derived from the defined filter bank and is suitable for detecting elliptical structures, such as cells and cell nuclei, at different orientations.
• the response of the filter bank can be interpreted as the maximum likelihood of a given pixel to be nuclei. The response is maximum in the center, and close to zero near the borders.
• a mapping function, p Gabor : R— > [0,1] is defined from the response of the filter bank I Gabor .
• the mapping function p Gabor is constructed so that it can be interpreted as a likelihood function that captures relevant morphological information from the given filter bank.
• Images of DAPI channels in cell-based tissue comprise rich morphological nuclei information.
• the morphological information a DAPI image provides can be used with other image transformations.
• the DAPI channel is used as a source of morphological information.
• the DAPI image is preprocesseed by applying morphological operations to the image, such as erosion and dilation.
• a function 34D, p j : R ⁇ [0,1] is defined, which maps the intensity values to probabilities (FIG. 3D).
• a parametric sigmoid function is defined to map the image intensity values to probability values.
• an estimate (FIG. 3C) is also generated based on a gradient segmentation algorithm 34C.
• the gradient segmentation is based on the magnitude of the gradient and has a maximum response at the border of the object, (e.g. in this example, the membrane of the nucleus) and a minimum response inside the object (e.g. nucleus). This is a penalty or distinguishing element that is used to separate the nuclei.
• the information of the gradient is complementary to the probability maps.
• seed points are also determined (FIG. 4B).
• the seed points I Seeds are located at the local maximum probabilities that are, in this example, normally at the center of the nuclei, and they are defined as Seed Seed
• the single connected regions c. are estimated from the probability map I CBS by applying the watershed transform to the distance transform image, derived from binary volume B CBS .
• FIG. 4B shows the detected nuclei center points I Seeds . These are used as a set 42 to impose regions of local minimum in the watershed algorithm. The background is also excluded, so regions
• the background is estimated from the combined stronger probability map generated from the individual weaker segmentations.
• a watershed step 44 is then carried out by applying the morphological constraints to the weighted image (FIG. 4C), as illustrated below:
• FIG. 4C is the local constraints image, notice that the nuclei center is black since it corresponds to a local minimum, and the cell borders are brighter since they correspond to local maximum.
• FIG. 4D presents the detected nuclei regions
• FIG. 4E presents the final segmentation after merging those regions that correspond to the same nuclei Example
• FIG. 5A-5D show a DAPI image corresponding to the xenograft model.
• FIG. 5A and 5B are the original and the segmented image, respectively.
• FIG. 6C and 6D show of the original image and the segmentation, respectively.
• the variations in shape, size, and appearance shown are due to non uniformity of the fluorescent dye.
• the segmentation results from the method are highly accurate and consistent, even in images where the cell nuclei are crowded, overlapping and frequently touching. As shown in FIG. 6, cells 50, 36, and 37 are clear distinguishable.

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Multimedia (AREA)
• Biomedical Technology (AREA)
• General Health & Medical Sciences (AREA)
• Molecular Biology (AREA)
• Life Sciences & Earth Sciences (AREA)
• Health & Medical Sciences (AREA)
• Probability & Statistics with Applications (AREA)
• Software Systems (AREA)
• Image Analysis (AREA)
• Image Processing (AREA)

Priority Applications (3)

Applications Claiming Priority (2)

Publications (2)

Family

ID=44760963

Family Applications (1)

Country Status (5)

Cited By (3)

Families Citing this family (39)

Family Cites Families (11)

• 2010
  • 2010-04-09 US US12/757,668 patent/US8300938B2/en not_active Expired - Fee Related
• 2011
  • 2011-04-06 WO PCT/SE2011/050407 patent/WO2011126442A2/en not_active Ceased
  • 2011-04-06 CN CN2011800179707A patent/CN102834846A/zh active Pending
  • 2011-04-06 JP JP2013503713A patent/JP5744177B2/ja not_active Expired - Fee Related
  • 2011-04-06 EP EP11766245.2A patent/EP2556491B1/en not_active Not-in-force

Non-Patent Citations (3)

Also Published As

Similar Documents

Legal Events