GB2497116A

GB2497116A - Segmentation of electromicrograph images

Info

Publication number: GB2497116A
Application number: GB1120681.0A
Authority: GB
Inventors: John Robert Maddison; Havard Danielsen
Original assignee: Institute for Medical Informatics
Current assignee: Institute for Medical Informatics
Priority date: 2011-12-01
Filing date: 2011-12-01
Publication date: 2013-06-05
Anticipated expiration: 2031-12-01
Also published as: GB2497116B; GB201120681D0

Abstract

A method of processing an electromicrograph image of at least 4 Megapixels including cells may use an analogue film image from an analogue camera 4 in an electron microscope 2 to obtain a high resolution image. The image may be digitised in digitiser 6 and is processed using a first segmentation operation at a first length scale to identify first segmentation regions of the image containing Hetrochromatin or Euchromatin. A second segmentation operation is then be carried out at a second length scale to identify second segmentation regions of the image within the first segmentation regions, the second segmentation regions containing nucleosomes or fibers. Following segmentation a number of metrics, such as area, form factor, shape factor, eccentricity and Hu pararameters may be used to characterize objects identified in the segmented regions The segmentation operation may be carried by means of a thresholding operation such as Otsu s method, the quadratic integral ratio(QIR) algorithm or the k-means cluster technique. The first segmentation method may be carried out on a decimated image and the second segmentation method may use Niblack s method of comparison of pixel values with a mean pixel value plus a first constant times the standard deviation of the pixel values less a second constant. The application may also include accepting zoom instruction to display the image at a plurality of magnifications.

Description

MICROGRAPHY

The invention relates to a method and apparatus for micrography.

Background Art

The use of electron micrographs is a widely used technique in the study of living matter, allowing images at cell level and at much finer length scales to image individual cell components. Such images may be used in research or in medicine.

Images may be displayed on a wide variety of imaging devices, typically computer screens. However, there is a growing interest in displaying such images on portable devices, such as mobile telephones or medium sized

tablets.

Images may be captured at a wide variety of magnifications and corresponding length scales and this in turn gives rise to a problem in relating images taken at the different magnifications. For example, a researcher may identify a cell that appears abnormal at one length scale and wish to "zoom in" on finer detail in that cell. However, it is not in general trivial to identify the same region on a separate micrograph taken at a different magnification.

Further, in order to capture images at different magnifications it is normally necessary to capture multiple images at the highest magnification and tile them to cover the area of the image captured at lower magnification.

However, tiling is very difficult in electron microscopes since it is not possible to simply move the sample and camera precisely enough to register the images together.

The digital cameras used in microscopy applications have limitations. In particular, the number of pixels is limited, typically not more than one megapixel. This is due to the demands of microscopy imaging applications.

There is thus a need for a means of displaying images on different length scales on a fixed or portable apparatus, such that features in the image at one length scale can be identified with those at another.

Increasingly, there is also a demand for automatically classifying and segmenting images, and there is accordingly also a need for method and apparatus to classify and segment images at different length scales.

Summary of Invention

According to the invention, there is provided a method according to claim 1.

It is preferred that the captured electromicrograph image has at least 4 megapixels, preferably at least 10 megapixels, further preferably at least 30 megapixels and most preferably at least 100 megapixels.

By capturing images on high resolution film, not digitally, and then digitising them, it is possible to obtain higher resolution images than using conventional digital microscopy cameras. This allows for the segmentation of the image at different length scales.

Preferably, the image may also be displayed at different magnifications. This allows a user to review the image and relate features at the different length is scales.

The method includes carrying out a first segmentation operation at a first length scale to identify first segmentation regions of the image containing Hetrochromatin or Euchromatin, and carrying out a second segmentation operation at a second length scale to identify second segmentation regions of the image within the first segmentation regions, the second segmentation regions containing nucleosomes or fibers.

The first segmentation operation may be carried out by carrying out a thresholding operation, and selecting as the first segmentation regions the regions resulting from the thresholding operation which exceed a determined number of pixels in size. The thresholding operation may be carried out using Otsu's method and the determined number of pixels may be in the range 500 pixels to 10000 pixels.

The second segmentation operation may be is carried out by comparing the pixel values with the mean pixel value plus a first constant (k) times the standard deviation of the pixel values less a second constant (c).

To carry out the method more efficiently, a decimation operation may be performed to obtain a decimated image having fewer than 4 megapixels from the electromicrograph image. This lower resolution image can then be used to carry out the first segmentation operation on the decimated image.

The captured image may be displayed on a display and zoomed to a plurality of different magnifications.

A plurality of metrics may be calculated characterising the second segmentation regions.

In another aspect, the invention also relates to a computer program product according to claim 10.

The invention also relates to electromicrography apparatus according to claim 11.

The rnicrography apparatus may in particular be a transmission electron microscope using an analogue film camera.

Brief Description of the Drawings

For a better understanding of the invention, an embodiment will now be described with references to the accompanying drawings, in which: Figure 1 is a schematic of apparatus according to an embodiment of the invention; Figure 2 is a histogram of a first image before norrnalisation; Figure 3 is a histogram of the first image after norrnalisation; Figure 4 is a histogram of a second image before normalisation; Figure 5 is a histogram of a second image after normalisation: Figure 6 is a micrograph before a first segmenting operation; Figure 7 is a plot indicating the masks identified by the first segmenting operation in the image of Figure 6; and Figure 8 is a micrograph indicating the regions identified by a second segmenting operation.

Detailed Description

Images of cells were captured with an electron microscope 2 using transmission electromicrography using analogue film in an analogue film camera 4. The images were digitised using digitiser 6 and passed to computer 8 with display 10. The images sizes obtained from the digitized analogue film are large, typically 11,000 by 19,000 pixels which is a 209 mega pixel Image. Such high pixel counts are far higher than available using microscopy imaging cameras which are typically still limited to approximately 1 megapixel due to their specialist nature and the requirement for high signal to noise ratios.

The use of the images from digitized analogue film allow for the very highly magnified objects within the cell such as the nucleosomes and fibers to be analyzed within the lower magnified areas of Heterochromatin and Euchromatin i.e. in context. This process is not possible from images that have much more limited resolution as obtained from a digital image without tiling images together. Tiling images is however a technically difficult due to extremely small movements required under the electron microscope.

The computer 8 effectively operates as a data analysis module for carrying out data analysis of the captured image, by running a computer program product recording the instructions for causing the computer to carry out the method as will now be described.

The images collected from the electron microscope 2 are output by the digitiser in a form that is not normalised. In order to allow the comparison of images from different cases, and within a case set, the images were normalised so the features of the images could be compared. To normalize the histogram of image intensity of pixels the histogram is stretched between the 5 and 95% points on the histogram so that the range of intensities varies between the 5% and 95% points.

For example, Figures 2 and 4 show the histograms of two images, in which the number of counts is plotted against intensity. Figures 3 and 5 show the histograms for the same images after normalisation. The normalised images are sufficiently standardised for the segmentation operations to work and for comparison to be possible.

Next, a first segmentation operation was carried out to identify the Heterochromatin and Euchromatin in the image, to identify the areas which may contain fibers and nucleosomes. This meant that when the image was further processed to identify fibers and nucleosomes it could be determined which area they came from.

To start this process, a decimated, original image was generated, i.e. an image at lower resolution such as 800x1300. Figure 6 shows an example of such an image. This was created by averaging the original high resolution image to remove high frequency detail not relevant to the first segmentation operation. This was done using a moving average -in this case a moving kernel of size 10 worked well. Other approaches or kernel sizes may be used as appropriate.

The image is then converted to a black and white image using a threshold level. The threshold level was automatically calculated using Otsu's method (http://en.wikiredia.org/wiki/Otsus method). Other approaches to thresholding are available and are mentioned below.

A morphological closing operator with a diameter of 15 pixels was then used so that the image identified areas, not fine detail, in the edges of the thresholded image. Other diameters or methods may be used.

To complete the first segmentation, only the objects with more than a predetermined number, 2000, pixels in the decimate image are retained as the results of the first segmentation. The resulting image mask is shown in Figure 7. The black areas are the first segmentation regions. The term "mask" is used to refer to such a black and white image showing the regions but not the original detail of the image.

Alternative predetermined numbers of pixels for the minimum size of objects retained, for example in the range 500 to 10 000, are possible.

The next stage is to carry out the second segmentation operation to determine nucleosomes and fibres. This is carried out in the first segmentation regions, which are the regions where the nucleosomes and fibres may be found, i.e. the regions in black in the image of Figure 7.

In order to allow for the image processing the high resolution image was split into manageable areas. Since the original image was the same high resolution image as used to form the decimate image used in the first segmentation, essentially exact matching of regions for the second segmentation and first segmentation is possible.

The nucleosomes and fibers were then segmented using the Niblack local thresholding method.

In Niblack's thresholding method the pixels are thresholded using the condition: pixel = (pixel > mean + k * standard_deviation -c) ? object: background where parameter 1 is the k value and parameter 2 is the c value.

The default value of k is 0.2 for bright objects and -0.2 for dark objects. Any other number than 0 will change the default value.

The c value is an offset with a default value of 0.

Further details of the original Niblack method with the default c value are available from Niblack, W (1986), "An introduction to Digital Image Processing" Prentice-Hall. In particular, note that the method is a local thresholding method, i.e. the image is thresholded using the image characteristics of those parts of the image within a certain distance of the pixels in question. Thus, the mean and standard deviation are calculated for the pixels within a window of radius r from the pixel in question.

An example of the resultant image mask showing the second segmentation regions at a fine scale is shown in Figure 8. It will be seen that a large amount of fine detail has been automatically segmented.

Alternative image processing is available, Instead of carrying out the thresholding for the first segmentation using the Otsu method alternative approaches may be used such as the Quadratic integral ratio (QIR) algorithm or the k-means cluster technique. A suitable approach originally used for an archaelogical method is described in "Thresholding: A Pixel-Level Image Processing Methodology Preprocessing Technique for an OCR System for the Brahmi Script" H. K. Anasuya Devi, available at http://www.ancient-asia-journal.com/article/view/aa.06335/25.

For the second segmentation, alternative algorithms are those proposed by Sauvola and Bernsen. An implementation of both algorithms is presently available at the website: http://pacific.mpi-cbg.de/wiki/index.php/Auto_Local_Threshold. These algorithms are examples of local thresholding algorithms, i.e. where the threshold is computed for each pixel according to the image characteristics within a window of radius r of that pixel.

Following the segmentation steps, the identified objects were characterised using a number of different metrics. The metrics are calculated using the objects segmented in the second segmentation steps, referred to below as the "mask". In the example, the calculated metrics were: Area Number of pixels within the mask Form Factor Diameter max ForniFactor = Diameter Form factor is the measure used to describe the shape in terms of the length of its minimum and maximum diameters, as opposed to shape factor in Section 3.2.3 which references object to a circle using perimeter and area measures. Diameter and Diameter max are the minimum and maximum diameters of the segmented cell.

Shape Factor Area ShapeFactor = 2.

Diameter flax Penmeter.( ) Shape factor is a parametric measure used to describe the circularity of an object where shape factor for a circle=1. Area, Perimeter and Diameter are object dimensions in pixels.

Eccentricity The eccentricity is the ratio of the distance between the foci of the ellipse and its major axis length.

Equivalent The equivalent diameter circle that has the same perimeter as Diameter the object.

Perimeter P = 1.41 Phrjs + iV I7ert IIoz PiteJv Perimeter P is the number of boundary pixels in the segmented N. object DrngoaIPte1 is the number of pixel on the perimeter of the object diagonally connected to neighbours and is the number of pixels connected to neighbouring pixels either vertically or horizontally.

HU Parameters Seven Hu Moments as described Calculated On The http:/Ien.wikipediaorq/wikitlmape moment are a set of Mask parameters that describe an object are calculated on the masked object.

Spatial moment calculated as Mpsum,( l(x,y)*x*y') where l(x,y) is the intensity of the pixel (x, y) From which the central moment is calculated where x0=M10/M00, y=Moi/Moo -coordinates of the gravity center And the normalised central moment ii= From which the seven Hu Moments are calculated hl=fl20+q02 h2=(q20_n02)2+4q1 i2 h3=(q30-3q12)2+ (3fl21-flos)2 h4=(rj3o+q12)2+ (q21+nos)2 hs=(qso-3fl12)Q13o+q12)[(q3o+ril2)2-3(fl21+flo3)2]+(3q2l-fl 03) ( 03) [3( q 30+11 12)2( fl21fl o) hs=(1120-1102)[(fl30+q12)2-(q21-'-flo3)2]4qh1(q3oq12)(q21qo3) 3r112)(r121 +flo3)[3(q30+n12)2(n21+fl03)2] HU Parameters Same calculation as HU Parameters Calculated On The Mask Calculated On The but on the pixel values of the object within the mask Gray Scale Image Within The Mask(GS) HU Parameters Same calculation as HU Parameters Calculated On The Mask Calculated On The but on the whole masked object Gray Scale Image On The Whole lmage(GS) Mean Within The.

Intens1tvm -Mask Mm Intensetwrfl is the mean intensity inside a segmented area and " is the number of pixels within the object, Intensity0 is the pixel intensity for individual pixels.

Stddev I A -((x. _x)2 Within The Mask _________ c='L

N

Standard deviation is c, where is the mean value, the sample value and N the number of samples.

The HU parameter values are proved to be invariants to the image scale, rotation, and reflection except the seventh one, whose sign is changed by reflection.

In the embodiment, all these metrics are calculated but alternative implementations may calculate a different set of metrics or none depending on the required result.

The image may be displayed on the display 10 under control of computer 8.

The image may be zoomed to different magnifications. The first and second segmentation regions may be displayed or not, under user control.

Although a single embodiment of the invention has been described, those skilled in the art will realise that alternative embodiments are possible.

For example, the "computer" may be any type of computer, and may in particular be a network of computers, including a mobile device such as a smart mobile telephone. This may allow a medical practioner to view the images conveniently.

Claims

<claim-text>Claims 1. A method of capturing and processing an electrornicrograph image of a sample containing cells, comprising: capturing an electromicrograph image of a sample having at least 4 megapixels; carrying out a first segmentation operation at a first length scale to identify first segmentation regions of the image containing Hetrochromatin or Euchromatin; carrying out a second segmentation operation at a second length scale to identify second segmentation regions of the image within the first segmentation regions, the second segmentation regions containing nucleosomes or fibers.</claim-text> <claim-text>2. A method according to claim 1 wherein capturing an electomicrograph image includes: recording the image on analogue film; digitising the image recorded on film.</claim-text> <claim-text>3. A method according to claim 1 or 2 wherein the first segmentation operation is carried out by: carrying out a thresholding operation, and selecting as the first segmentation regions the regions resulting from the thresholding operation which exceed a determined number of pixels in size - 4. A method according to claim 3 wherein the thresholding operation is carried out using Otsu's method.5. A method according to claim 3 or 4 wherein the determined number of pixels is in the range 500 pixels to 10 000 pixels.6. A method according to any preceding claim wherein the second segmentation operation is carried out by comparing the pixel values with the mean pixel value plus a first constant (k) times the standard deviation of the pixel values less a second constant (c), wherein the mean and standard deviation are the mean and standard deviation calculated for pixels within a predetermined radius of the pixel being compared.7 A method according to any preceding claim, further comprising carrying out a decimation operation to obtain a decimated image having fewer than 4 megapixels from the electromicrograph image, and carrying out the first segmentation operation on the decimated image.8. A method according to any preceding claim, further comprising displaying the electromicrograph image on a display and accepting zoom instructions on an input control to zoom the image to a plurality of different magnifications.9. A method according to any preceding claim further comprising calculating a plurality of metrics on the second segmentation regions.10. A computer program product arranged to: carry out a first segmentation operation on an electromicrograph image having at least 4 megapixels, the first segmentation operation being carried out at a first length scale to identify first segmentation regions of the image containing Hetrochromatin or Euchromatin; and to carry out a second segmentation operation at a second length scale to identify second segmentation regions of the image within the first segmentation regions, the second segmentation regions containing nucleosome or fibers.11. An electromicrography apparatus, comprising a transmission electron microscope for capturing an electromicrograph image of a sample having at least 4 megapixels; and a data analysis module having a data processor arranged to: carry out a first segmentation operation at a first length scale to identify first segmentation regions of the image containing Hetrochromatin or Euchromatin; and to carry out a second segmentation operation at a second length scale to identify second segmentation regions of the image within the first segmentation regions, the second segmentation regions containing nucleosome or fibers.12. An electromicrography apparatus according to claim 11 wherein the transmission electron microscope uses an analogue film camera.</claim-text>