WO2000067060A1 - Procedes et appareil pour une resolution en profondeur amelioree en microscopie, au moyen d'informations hors foyer - Google Patents

Procedes et appareil pour une resolution en profondeur amelioree en microscopie, au moyen d'informations hors foyer Download PDF

Info

Publication number
WO2000067060A1
WO2000067060A1 PCT/US2000/011548 US0011548W WO0067060A1 WO 2000067060 A1 WO2000067060 A1 WO 2000067060A1 US 0011548 W US0011548 W US 0011548W WO 0067060 A1 WO0067060 A1 WO 0067060A1
Authority
WO
WIPO (PCT)
Prior art keywords
sample
axis
focus data
along
focus
Prior art date
Application number
PCT/US2000/011548
Other languages
English (en)
Inventor
Calum E. Macaulay
Original Assignee
Digital Optical Imaging Corporation
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Digital Optical Imaging Corporation filed Critical Digital Optical Imaging Corporation
Priority to AU45001/00A priority Critical patent/AU4500100A/en
Publication of WO2000067060A1 publication Critical patent/WO2000067060A1/fr

Links

Classifications

    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/0004—Microscopes specially adapted for specific applications
    • G02B21/002—Scanning microscopes
    • G02B21/0024—Confocal scanning microscopes (CSOMs) or confocal "macroscopes"; Accessories which are not restricted to use with CSOMs, e.g. sample holders
    • G02B21/0052—Optical details of the image generation
    • G02B21/006—Optical details of the image generation focusing arrangements; selection of the plane to be imaged
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/0004—Microscopes specially adapted for specific applications
    • G02B21/002—Scanning microscopes
    • G02B21/0024—Confocal scanning microscopes (CSOMs) or confocal "macroscopes"; Accessories which are not restricted to use with CSOMs, e.g. sample holders
    • G02B21/0052—Optical details of the image generation
    • G02B21/0072—Optical details of the image generation details concerning resolution or correction, including general design of CSOM objectives
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/0004—Microscopes specially adapted for specific applications
    • G02B21/002—Scanning microscopes
    • G02B21/0024—Confocal scanning microscopes (CSOMs) or confocal "macroscopes"; Accessories which are not restricted to use with CSOMs, e.g. sample holders
    • G02B21/008—Details of detection or image processing, including general computer control

Definitions

  • the field of the present invention is confocal microscopy.
  • Microscopy is used to produce magnified representations of both dynamic and stationary objects or samples; microscopes magnify small things and make them easier to see.
  • There are many different modes of microscopy such as brightfield microscopy, darkfield microscopy, phase contrast microscopy, fluorescence microscopy, reflectance or reflected light microscopy and confocal microscopy. All of these forms of microscopy deliver illumination light in a controlled fashion to the sample and collect as much of the light containing the desired information about the sample as possible. Typically, this is accomplished using Kohler illumination in any of reflectance microscopy, transmission microscopy or epifluorescence microscopy.
  • diaphragms are placed in at least two locations. First, a diaphragm is placed in the conjugate image plane of the sample, a location which permits control of the size of the illuminated area of the sample. Second, a diaphragm is placed in the conjugate image plane of the aperture diaphragm of the objective lens(es) (this location is also a conjugate image plane of the aperture diaphragm of the condensor lens(es)), a location which 2
  • any of the diaphragms can be a simple iris (for example, for brightfield microscopy and epillumination fluorescence microscopy), but the diaphragms can also be more complex (for example, in darkfield microscopy, where the diaphragms may comprise cutout rings of different diameters).
  • microscope 2 comprises a light source 4 that emits a plurality of light rays, which have been divided into first light rays 6, second light rays 8 and third light rays 10.
  • the light rays are transmitted along an illumination light path from light source 4 through light source lens 12, adjustable iris field diaphragm 14 and condensor lenses 16.
  • An adjustable iris aperture diaphragm (condensor) 18 can be disposed between upstream and downstream condensor lenses 16.
  • the light then contacts, or impinges upon, sample 20 and then proceeds to pass through objective lenses 22, which objective lenses can comprise an aperture diaphragm (objective) 24 spaced between the objective lenses 22, and then the light rays proceed to a light detector 26.
  • the angle of illumination of the sample can be controlled by modulating the light as it passes through conjugate image planes of the aperture diaphragm of the objective lens, which planes can be found, for example, at light source 4 and the upstream aperture diaphragm 18 in Figure 1, while the location and/or area of illumination of the sample can be controlled by modulating light as it passes through a conjugate image plane of the sample, which plane corresponds to the adjustable iris field diaphragm 14 in Figure 1.
  • microscopy is confocal microscopy, in which one or more discreet aperture spots are illuminated in the object plane of the microscope from which transmitted, reflected or fluorescent light is then relayed for observation through conjugate apertures in the image plane.
  • confocal microscopy can result in spatial resolution about 1.3 times better than the optimum resolution obtainable by conventional light microscopy. See, e.g., U.S. Patent No. 5,587,832.
  • confocal microscopy can reduce the interference of stray, out-of-focus light from an observed specimen above or below the focal plane, and can permit optical sectioning of tissue as well as high-resolution 3-D reconstruction of the tissue. The technique can effectively resolve individual cells and living tissue without staining.
  • Confocal microscopy can be performed using mechanical translation of the specimen with fixed optics, using a fixed specimen and scanning beams manipulated by special rotating aperture disks, or a spatial light modulator (SLM).
  • SLM spatial light modulator
  • the special rotating aperture disks often called Nipkow disks, typically comprise a plurality of apertures, but only one aperture at a time is used for confocal scanning.
  • Still other known confocal scanning systems have used a laser beam rastered with rotating mirrors to scan a specimen or a laser beam that scans a slit rather than a spot; such slit scanning increases imaging speed relative to rotating aperture disks but slightly degrades resolution.
  • the use of spatial light modulators permits control of either or both of the angle(s) of the light and location of the light, and can provide high speed confocal scanning without the loss of resolution that accompanies slit scanning instead of spot scanning.
  • United States patent application serial No. 09/179,185 entitled Apparatus And Methods Relating To Spatially Light Modulated Microscopy; U.S. Patent No. 5,867,251.
  • Confocal microscopy does not utilize a significant portion of the light emanating from the spot on the sample that is under investigation, and thus has unnecessarily limited resolution in both the x-y plane (sideways) and in the z-direction (up and down, or depth), and an unnecessarily limited signal to noise ratio.
  • Confocal microscopy does not utilize a significant portion of the light emanating from the spot on the sample that is under investigation, and thus has unnecessarily limited resolution in both the x-y plane (sideways) and in the z-direction (up and down, or depth), and an unnecessarily limited signal to noise ratio.
  • the present invention provides these and other advantages.
  • the present invention provides apparatus and methods that improve the depth resolution of confocal microscopy images.
  • the present invention can be applied to all of reflectance microscopy, transmission microscopy and fluorescence microscopy. 4
  • the present invention comprises utilizing out-of-focus information from within the focal plane of interest (from the x-y direction) and/or from planes above and below the focal plane of interest (from the z-direction).
  • the present invention takes advantage of the observation that in confocal microscopy the intensity of the light emanating from the illumination spot of the sample falls off or decreases in a regular fashion as the distance from the illumination spot increases.
  • the point spread function (PSF) of the emanating light for a confocal, cylindrically symmetric lens system falls off approximately as sinc ⁇ 2(z) for the singularly illuminated spot, or central illumination pixel, in the vicinity of the focal plane.
  • This PSF in the x-y plane is a function of depth (i.e., of the z-position).
  • a reflective surface or other light-emanating surface such as a fluorescent surface or transmissive surface
  • the PSF formed by a confocal microscope results in "out-of-focus” information in, above and below the focal plane; this "out-of-focus” information can be measured.
  • the present invention provides confocal microscopes comprising a light detection and analysis system, the system comprising a light detector disposed downstream from a sample in a conjugate image plane of the sample.
  • the detector comprises a central detection pixel positioned to detect and measure in-focus light emanating from a discrete illumination pixel of the sample to provide in-focus data and at least one adjacent detection pixel in an x-y plane relative to the sample that is positioned to independently detect and measure out-of-focus light emanating from the discrete illumination pixel of the sample in the x-y plane. This provides out-of-focus data in the x-y plane.
  • the system further comprises a controller operably connected to the detector and containing computer implemented programming that compiles and combines or convolves the in-focus data and the out-of-focus data to enhance resolution of the discrete illumination pixel of the sample when compared to a resolution obtained without using the out-of-focus data.
  • the resolution is enhanced in the z-direction by at least about 5% or 10%, preferably by at least 25-100%, and in the x-y plane by at least about 10%, and further preferably by at least about 15-40%).
  • the detector comprises a plurality of adjacent detection pixels that surround the central detection pixel and that independently detect and measure out-of-focus light emanating from the discrete illumination pixel of the sample, and the central detection pixel and the at least one adjacent detection pixel can abut each other.
  • the controller may fit the out-of-focus data in the x-y plane according to a 2D Gaussian distribution or according to other suitable fitting functions.
  • the detector can be movably connected to the sample along a z-axis of the sample such that movement of the detector relative to the sample permits the detector to detect and measure in-focus data from a focal plane of the sample along the z-axis and out-of-focus data from above or below the focal plane along the z-axis and from the x-y direction within each of such planes.
  • the controller can further contain computer implemented programming that compiles and combines the in-focus data from along the z-axis and the out-of-focus data from along the z-axis to enhance resolution of the discrete illumination pixel of the sample when compared to a resolution obtained ⁇ . ithout using the out-of- focus data from along the z-axis.
  • the microscope further comprises a spatial light modulator disposed upstream of the sample in a conjugate image plane of the sample and computer implemented programming that causes the spatial light modulator to simultaneously form a plurality of the illumination spots that illuminate a plurality of discrete illumination pixels of the sample and to provide sequential complementary patterns of the spots.
  • the spatial light modulator can be disposed upstream of the sample in a conjugate image plane of the sample and can be operated to selectively alternate between brightfield microscopy and confocal microscopy. 6
  • the microscope further comprises a reference mirror disposed in a conjugate image plane of the sample, the reference mirror movably connected to the detector along a z-axis of the mirror such that movement of the reference mirror relative to the detector permits the detector to detect and measure in-focus data from a focal plane of the reference mirror along the z-axis and out-of-focus data from above and below the focal plane along the z-axis, and wherein the controller contains computer implemented programming that compiles the in-focus data from along the z- axis of the reference mirror and the out-of-focus data from along the z-axis of the reference mirror to provide a reference stack of reference mirror images, and combines and convolves or compiles or otherwise compares the reference stack with the measurements of in-focus data and out-of-focus data from the z-axis of the sample, to thereby determine the location of the focal plane of the sample and thus enhance resolution of the discrete illumination pixel of the sample when compared to a resolution obtained without using the out-of-focus
  • the invention provides a light detection and analysis system, the system comprising a light detector disposed downstream from a sample in a conjugate image plane of the sample, wherein the detector is movably connected to the sample along a z-axis of the sample such that movement of the detector relative to the sample permits the detector to detect and measure in-focus data from a focal plane of the sample along the z-axis and out-of-focus data from above and below the focal plane along the z-axis, the system further comprising a controller operably connected to the detector and containing computer implemented programming that compiles and combines the in- focus data from along the z-axis and the out-of-focus data from along the z-axis to enhance resolution of the discrete illumination pixel of the sample when compared to a resolution obtained without using the out-of-focus data from along the z-axis.
  • the detector further comprises a central detection pixel and at least one adjacent detection pixel
  • the controller contains computer implemented programming that compiles and combines the in-focus data and the out-of-focus data to enhance resolution of the discrete illumination pixel of the sample when compared to a resolution obtained without using the out-of-focus data.
  • the microscope can be a reflectance microscope, transmission microscope, fluorescence microscope or other confocal microscope.
  • the present invention provides a controller suitable for operable connection to a confocal microscope, wherein the controller comprises a digital light detector disposed downstream from a sample in a conjugate image plane of the sample, the detector comprising a central detection pixel positioned to detect and measure in-focus light emanating from a discrete illumination pixel of the sample to provide in- focus data and at least one adjacent detection pixel in an x-y plane relative to the sample and positioned to independently detect and measure out-of-focus light emanating from the discrete illumination pixel of the sample in the x-y plane to provide out-of-focus data in the x-y plane.
  • the controller comprises a digital light detector disposed downstream from a sample in a conjugate image plane of the sample, the detector comprising a central detection pixel positioned to detect and measure in-focus light emanating from a discrete illumination pixel of the sample to provide in- focus data and at least one adjacent detection pixel in an x-y plane relative to the sample and positioned to independently detect and measure out-of-
  • the controller further contains computer implemented programming that compiles and combines the in-focus data and the out-of-focus data to enhance resolution of the discrete illumination pixel of the sample when compared to a resolution obtained without using the out-of-focus data.
  • the controller fits the out-of-focus data in the x-y plane according to a 2D Gaussian distribution.
  • the detector of the microscope under the control of such controller can be movably connected to the sample along a z-axis of the sample such that movement of the detector relative to the sample permits the detector to detect and measure in-focus data from a focal plane of the sample along the z-axis and out-of-focus data from above and below the focal plane along the z-axis.
  • the controller preferably further contains computer implemented programming that compiles and combines or otherwise compares the in-focus data from along the z- axis and the out-of-focus data from along the z-axis to enhance resolution of the discrete illumination pixel of the sample when compared to a resolution obtained without using the out-of-focus data from along the z-axis.
  • the controller may also, if desired, fit information where the detector is movably connected along a z-axis to a reference mirror disposed in a conjugate image plane of the sample such that movement of the reference mirror relative to the detector permits the detector to detect and measure in-focus data from a focal plane of the reference mirror along the z-axis and out-of-focus data from above and below the focal plane along the z-axis, wherein the controller further contains computer implemented programming that compiles the in-focus data from along the z-axis of the reference mirror and the out-of-focus data from along the z-axis of the reference mirror to provide a reference stack of reference mirror images, and convolves or otherwise compares the reference stack with the measurements of in-focus data and out-of-focus data from the z- axis of the sample, to thereby determine the location of the focal plane of the sample and thus enhance resolution of the discrete illumination pixel of the sample when compared to a resolution obtained without using the out-of-focus data from along the z-axis
  • the present invention provides a confocal microscope comprising means for detecting and measuring out-of-focus light emanating from a discrete illumination pixel of a sample in at least one of an x-y plane and a z-axis to provide a measurement of out-of-focus light, and means for combining the measurement of the out-of-focus light with a measurement of in-focus light emanating from the discrete illumination pixel of the sample, to provide an enhanced resolution of the discrete illumination pixel of the sample when compared to a resolution obtained without using the out-of-focus data.
  • the means for detecting can detect and measure out- of-focus light from both the x-y plane and the z-axis, and the means for combining can combine the measurement of light from both the x-y plane and the z-axis.
  • the microscope can additionally comprise means for providing a reference stack of reference images along the z-axis, and means for convolving the reference stack with the measurements of in-focus data and out-of-focus data from the z-axis of the sample, to thereby determine the location of the focal plane of the sample and thus enhance resolution of the discrete illumination pixel of the sample when compared to a resolution obtained without using the out-of-focus data from along the z-axis.
  • the present invention provides methods of enhancing resolution of a discrete illumination pixel of a sample using confocal microscopy, comprising detecting and measuring in-focus light emanating from the discrete illumination pixel to provide in-focus data, detecting and measuring out-of-focus light emanating from the discrete illumination pixel in an x-y plane of the sample to provide out-of-focus data in the x-y plane, and compiling and combining the in-focus data 9
  • the out-of-focus data in the x-y plane can be fitted according to a 2D Gaussian distribution or according to other suitable fitting methods.
  • the method further comprises detecting and measuring in-focus data from a focal plane of the sample along the z-axis of the sample and out-of-focus data from above or below the focal plane along the z-axis. Additionally, the detecting and measuring of out-of-focus data can be from both above and below the focal plane along the z-axis.
  • the methods can also further comprise compiling and combining the in-focus data from along the z-axis and the out-of-focus data from along the z-axis to enhance resolution of the discrete illumination pixel of the sample when compared to a resolution obtained without using the out-of-focus data from along the z- axis, and simultaneously forming a plurality of the illumination spots on the sample to provide a plurality of discrete illumination pixels of the sample, and simultaneously detecting and measuring the light emanating from the plurality of discrete illumination pixels and compiling and combining the in-focus and out-of-focus data from the illumination spots.
  • the methods further comprise providing sequential complementary patterns of the illumination spots until the entire surface of the sample has been illuminated.
  • the methods can additionally comprise selectively alternating between brightfield microscopy and confocal microscopy.
  • the methods can also comprise detecting and measuring in-focus light reflecting from a discrete illumination pixel of a reference mirror disposed in a conjugate image plane of the sample, the in-focus light being from a focal plane of the reference mirror along the z-axis, detecting and measuring out-of-focus light reflecting from the discrete illumination pixel of the reference mirror, the out-of-focus light being from above and below the focal plane along the z-axis, compiling the in-focus data from along the z-axis of the reference mirror and the out-of-focus data from along the z-axis of the reference mirror to provide a reference stack of reference mirror images, convolving the reference stack with the measurements of in-focus data and out-of-focus data from the z- axis of the sample, and determining
  • the present invention provides methods of enhancing resolution of a discrete illumination pixel of a sample during microscopy comprising detecting and measuring in-focus data from a focal plane of the sample along the z-axis of the sample, detecting and measuring out-of-focus data from above and below the focal plane along the z-axis of the sample, and compiling and combining the in-focus data from along the z-axis and the out-of-focus data from along the z-axis to enhance resolution of the discrete illumination pixel of the sample when compared to a resolution obtained without using the out-of-focus data from along the z-axis.
  • the methods can further comprise detecting and measuring out-of-focus light emanating from the discrete illumination pixel in an x-y plane of the sample to provide out-of-focus data in the x-y plane, and compiling and combining the in-focus data and the out-of-focus data in the x-y plane to enhance resolution of the discrete illumination pixel of the sample when compared to a resolution obtained without using the out-of-focus data in the x-y plane, preferably using a 2D Gaussian distribution and simultaneously forming a plurality of the illumination spots on the sample to provide a plurality of discrete illumination pixels of the sample, and simultaneously detecting and measuring the light emanating from the plurality of discrete illumination pixels and compiling and combining the in-focus and out-of-focus data from the illumination spots.
  • the methods comprise providing sequential complementary patterns of the illumination spots until the entire surface of the sample has been illuminated and can include selectively alternating between brightfield microscopy and confocal microscopy.
  • the methods can further comprise detecting and measuring in-focus light reflecting from a discrete illumination pixel of a reference mirror disposed in a conjugate image plane of the sample, the in-focus light being from a focal plane of the reference mirror along the z-axis , detecting and measuring out-of-focus light reflecting from the discrete illumination pixel of the reference mirror, the out-of-focus 11
  • Figure 1 provides a schematic view and an expanded schematic view of a conventional transmission light microscope using Kohler illumination.
  • Figure 2 provides a graph depicting the improvement in resolution achieved by using out-of-focus information.
  • Figure 3 provides a graph depicting the improvement in the rejection out- of-focus information as well as the increased signal to noise of the fitted data compared to the central pixel data.
  • Figure 4 provides a graph showing the measured locations of in-focus positions for a grid of spots measured across a tilted mirror using a conventional confocal method (i.e.. without using out-of-focus information).
  • Figure 5 provides a graph showing the measured locations of in-focus positions for a grid of spots measured using out-of-focus information.
  • Figure 6 provides a graph showing the measured locations of in-focus positions for a grid of spots measured using out-of-focus information. 12
  • the present invention provides apparatus and methods that improve the depth resolution and signal to noise ratio of confocal microscopy images, including reflectance, transmission and fluorescence microscopy.
  • the present invention utilizes out-of-focus information from the x-y direction of the focal plane as well as information from planes above and below the focal plane (from the z-direction).
  • confocal microscopy the intensity of the light emanating from the illumination spot of the sample falls off in a regular fashion as the distance from the illumination spot increases, both in the x-y plane and in the z-direction.
  • the interaction of a light-emanating material in a sample and the PSF formed by a confocal microscope results in "out-of-focus" information in, above and below the focal plane.
  • This "out-of-focus" information can be measured and used.
  • the resolution can be improved or enhanced along both the z- direction and the x-y plane when compared to the resolution that is achieved without the use of the out-of-focus information.
  • the resolution is enhanced in the z- direction by at least about 10%>, preferably by at least 25-100%>, and in the x-y plane by at least about 10%, and further preferably by at least about 15-40%.
  • a larger number of the photons in the system are used, thus improving the signal to noise ratio.
  • incorporation of such information can also improve depth resolution and otherwise help correct for aberrations in the optical system of a microscope.
  • the reflective surface or other sample surface is large with respect to the x-y resolution, that the reflectivity or other light emanation does not vary substantially relative to the spot size on the surface in the x and y directions, and that the surface to be imaged is approximately normal to the optical axis.
  • a brightfield image of the region of interest can be used to produce a reflectivity or light emanation map of the surface in question.
  • This map is used to assign weights to x-y pixels, preferably all such pixels, to such that each respective x-y position contributes correctly to the convolution or other combination calculation. For example, brighter pixels from the reflectivity map can be assigned smaller weights, while darker pixels from the reflectivity map can be assigned higher weights.
  • the surface is not normal to the optical axis.
  • This restriction for normal surface is due to the fact that the reference stack of images gathered in the z- direction - for example the stack of images collected while illuminating the mirror - contains information about the 3-dimensional interaction of the PSF off a surface normal to the optical axis.
  • additional reference stacks can be collected for various angles of incidence between the reference mirror and the light beam. These reference stacks are then used with the image stack obtained from the sample (again for various offsets along the z-axis).
  • the reference stack and offset position for which the convolution calculation yields the highest result give a measure of the depth and surface slope of the target position. Definitions.
  • a “discrete illumination pixel” or spot is an area, typically on or in a sample but potentially also on or in a reference mirror or other desired surface or material, that is illuminated by an illumination light in a confocal microscope wherein such spot is the only such spot illuminated by the confocal microscope or wherein such spot is sufficiently separated from other illumination pixels such that the discrete illumination pixel is capable of being detected by a central detection pixel without undue 14
  • the discrete illumination spot is capable of being detected by the central detection pixel and at least one detection pixel adjacent to the central detection pixel without undue interference from the other illumination pixels.
  • a "central detection pixel” is a spot in a pixelated detector, such as a charge coupled device (CCD), charge injection device (CID), or other detector capable of detecting or distinguishing individual detection locations within the detector corresponding to an illumination pixel within a sample.
  • the spot comprises a single detection element or pixel in the detector, but it can also comprise a plurality of detection elements or pixels.
  • the light from the discrete illumination pixel of the sample is directly conveyed via an optical path to the central detection pixel, which light path may include one or more lenses, mirrors, beam splitters or other optical elements.
  • the light from the illumination pixel of the sample can also be conveyed to the central detection pixel via electronic pathways or other pathways; preferably such transmission does not introduce any non-distinguishable artifacts into the data from the illumination pixel.
  • the central detection pixel detects in-focus light from the illumination pixel.
  • a detection pixel is adjacent to the central detection pixel when it is capable of receiving significant "stray" light from the illumination pixel; preferably a detection pixel abuts the central detection pixel or abuts a detection pixel that in turn abuts the central detection pixel.
  • a detection pixel detects out-of- focus light from the illumination pixel.
  • the adjacent detection pixel is able to independently detect and measure out-of-focus light emanating from the central illumination pixel when it can deliver different information about the out-of-focus light than the information that is delivered by the central detection pixel about the in-focus light.
  • the pixels herein are often described as capable of detecting and measuring light or other information; the actual measuring or analysis of the light may take place in a processor or other device that is not physically located within the pixel.
  • In-focus indicates light, data derived from the light, or other signal or information, emanating from a discrete illumination pixel to a corresponding central detection pixel.
  • Out-of-focus indicates light, data derived from the light, or other signal 15
  • the "out-of-focus” information can be either "out-of-focus” in the x-y plane of the sample (generally, sideways or laterally), or in the z-direction of the sample (generally, up and down, or vertical).
  • a “spatial light modulator” is a device that is able to selectively modulate light.
  • spatial light modulators are disposed in the light path of a microscope.
  • the spatial light modulator comprises an array of individual light transmission pixels, which are a plurality of spots that have transmissive characteristics such that they either transmit or pass the light along the light path or block the light and prevent it from continuing along the light path (for example, by absorbing the light or by reflecting it out of the light path).
  • Such pixelated arrays are well known in the art, having also been referred to as a multiple pattern aperture array, and can be formed by an array of ferroelectric liquid crystal devices, by a digital micromirror device, or by electrostatic microshutters. See U.S. Patent No.
  • the "illumination light path” is the light path from a light source to a sample
  • a “detection light path” is the light path for light emanating from a sample to a detector.
  • Light emanating from a sample includes light that reflects from a sample, is transmitted through a sample, or is created within the sample, for example, fluorescent light that is created within a sample pursuant to excitation with an appropriate wavelength of light (typically UV or blue light).
  • a "conjugate image plane of the sample” is a plane in either the illumination light path or the detection light path where an image of the sample is recreated.
  • the light detector(s) is typically located in one such site in the detection light path.
  • the conjugate image planes of the sample defines locations that can control the size and location of spots on the sample that are illuminated and/or detected (depending upon whether the conjugate plane is in the illumination light path or the detection light path).
  • the image plane of the sample is the plane wherein the sample is located, although the image plane of the sample can be greater or smaller than the size of the actual sample if either a plurality of light paths are provided or if the illumination area is greater or smaller than the size of the sample.
  • a "controller” is a device that is capable of controlling a spatial light modulator, a detector, including pixels within a detector, or other elements of the apparatus and methods of the present invention.
  • the controller can control the transmissive characteristics of the pixels in a spatial light modulator, control the on/off status of pixels of a pixelated light detector (such as a charge coupled device (CCD) or charge injection device (CID)), and/or compile data obtained from the detector, including using such data to make or reconstruct images or as feedback to control an upstream spatial light modulator.
  • a controller comprises one or more computers or other devices comprising a central processing unit (CPU) and contains computer-implemented programming that directs the controller, or from which the controller directs other devices, to perform certain functions or actions, such as those functions and actions described herein.
  • Controllers and appropriate computer- implemented programming associated with such controllers, are either well known in the art or well within the skill of a skilled artisan in view of the present disclosure, and thus selection or creation of a desirable controller or computer-implemented programming for a particular aspect of the present invention is within the scope of the art in view of the present disclosure.
  • Upstream and downstream are used in their traditional sense wherein upstream indicates that a given device is closer to a light source, while downstream indicates that a given object is farther away from a light source.
  • the terms set forth in this application are not to be interpreted in the claims as indicating a "means plus function” relationship unless the word “means” is specifically recited in a claim, and are to be interpreted in the claims as indicating a “means plus function” relationship where the word “means” is specifically recited in a claim.
  • the terms set forth in this application are not to be interpreted in method or process claims as indicating a "step plus function” relationship unless the word "step” 17
  • the resulting set of data provides a three- dimensional intensity map above, below, and around the in focus regions of the various surfaces to be detected and localized for each illuminated spot.
  • ⁇ x , ⁇ y are the spatial locations in the x-y plane which maximize the fit of the equation to the observed intensities
  • ⁇ x , ⁇ y are the widths in the x and the y directions that allow for the best fit of the measured intensities by a 2D Gaussian function
  • A is the scaling value which results in the best fit of the 2D Gaussian function to the data. Determining "best fit" for the given situation will be apparent to a person of ordinary skill in the art in view of the present application.
  • best fit examples include least squares and weighted least squares.
  • Other fitting functions can also be used.
  • some preferred functions for the optical analysis of lens systems include (sin(x)/x) 2 or a function based on a Bessel function of the first kind, 2J,[x*sin(q)j7(x*sin(q)).
  • a suitable routine to fit Gaussian curves to image data is the Levenberg-Marquardt Method, from "Numerical Recipes in C", 2 nd Edition, William H. Press et. al., Cambridge University Press, 1992, pp. 683-687.
  • the intensity of the pixel or integrated intensities of a disk of pixels centered on the (x-y) position of the illumination spot is designated as the measured confocal value at that location (x-y, z).
  • the measured confocal value is preferably either A/2 ⁇ x ⁇ y or A/2 ⁇ x 2 ⁇ y 2 . Both of these values have the advantage of being calculated from all of the reflected light collected by the optical system of the microscope, and not just from just the in focus light as is done in a conventional confocal microscope. Thus, because more light (photons) is (are) used to estimate the confocal value, the error (Poisson noise) associated with the measured value is less.
  • the z-axis position that results in the most significant measured value represents the position at which the illuminated spot is in focus.
  • the use of A/2 ⁇ x ⁇ y or A/2 ⁇ x 2 ⁇ y 2 as the measured value has the benefit of being much more dependent upon the sample position along the z-axis as seen in graph A and B.
  • using A/2 ⁇ x ⁇ _ or A/2 ⁇ ⁇ 2 ⁇ y 2 results in a confocal system with much less depth of field than an optically equivalent confocal microscope.
  • this embodiment can be extended to other approaches to confocal microscopy.
  • it can be used for fluorescence confocal microscopy by measuring the system's response to the illumination of a point fluorescence source at a variety of in-focus and out-of-focus positions along and off the optical axis of the system, recording the illumination spot intensity as well as the intensities in the surrounding areas, finding the functions which fit the intensity responses at the various locations, and building a weighted sum of these functions. This weighted sum of functions can, if desired, then be 19
  • confocal transmission microscopy can be effected by measuring the system's response to the illumination of a point light source at a variety of in-focus and out-of-focus positions along and off the optical axis of the system, and then proceeding as set forth above.
  • the present invention can be implemented by using a single illumination spot (or a plurality of discrete illumination spots) to illuminate a single x-y location on an optically flat plane mirror, and measure the signal generated at this location and in the surrounding region. Then, collect a stack of images like the stack of images collected in the methods described above and elsewhere herein, for various z-positions (preferably at regular or otherwise identifiable increments), both above and below the focal plane for the mirror. This image stack equates to a representation of the full 3-dimensional PSF as it interacts with the reflective surface of the mirror. Using the same x-y position, illuminate the object of interest. Again collect a stack of images, at the same increments of z position.
  • the offset at which this comparison returns the highest result, or otherwise most significant result is a measure of the location along the z-axis of the x-y position on the surface of the object.
  • the measurement process may take place in parallel.
  • the illumination and detection of the sample and mirror are controlled by a digital micromirror device (DMD) or other digital spatial light modulator, which DMD can rapidly switch the light back and forth between the mirror and the sample.
  • DMD digital micromirror device
  • An optically flat mirror was illuminated with an array of well- spaced points of light in the x-y plane to provide a plurality of discrete illumination pixels.
  • ⁇ v ⁇ _ were the spatial locations in the x-y plane that maximized the fit of the equation to the observed intensities, ⁇ x , ⁇ ; were the widths in the x and the y directions which allow for the best fit of the measured intensities by a 2D Gaussian function and A is the scaling value which results in the best fit of the 2D Gaussian function to the data.
  • the measured confocal value was also fitted using A/2 ⁇ x 2 ⁇ y 2 in the equation.
  • Figure 2 depicts the results obtained using A/2 ⁇ x ⁇ y
  • Figure 3 depicts the results obtained using A/2 ⁇ x 2 ⁇ y 2 .
  • Example 2 1) A plurality of discrete illumination pixels were used to illuminate a corresponding plurality of x-y locations on a tilted, optically flat plane mirror, and the signal generated at such locations and in the surrounding region were measured.
  • FIG. 2 A stack of images like the stack collected in Example 1 was collected, for various z-positions (at regular increments), both above and below the focal plane for the mirror.
  • This mirror stack was a representation of the full 3-dimensional PSF as it interacted with the reflective surface of the mirror.
  • Figure 4 provides a graph showing the measured locations of in-focus positions for a grid of spots measured across the tilted mirror without using out-of-focus information.
  • Figures 5 and 6 provide graphs showing the measured locations of in-focus positions for a grid of spots measured using out-of-focus information; Figures 5 and 6 were fitted using a 2 ⁇ x ⁇ y or a 2 ⁇ x 2 ⁇ y 2 , respectively, similar to the manner followed in example 1 above.
  • the fitted values Figures 5 and 6) have much less scatter and fall much closer to a simple plane than do the single pixel values in Figure 4 for in-focus positions found without the use of out-of-focus information.

Landscapes

  • Physics & Mathematics (AREA)
  • Chemical & Material Sciences (AREA)
  • Analytical Chemistry (AREA)
  • General Physics & Mathematics (AREA)
  • Optics & Photonics (AREA)
  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Engineering & Computer Science (AREA)
  • Microscoopes, Condenser (AREA)

Abstract

L'invention concerne un appareil et des procédés qui améliorent la résolution en profondeur d'images microscopiques confocales au moyen d'informations hors foyer provenant du plan focal concerné (sens x-y) et/ou des plans au-dessus et au-dessous du plan focal concerné (sens z). L'interaction (a) d'une surface réfléchissante ou d'un autre matériau duquel se dégage de la lumière et (b) le point d'étalement ponctuel (PSF) formé par un microscope confocal, induit la production d'informations « hors foyer » dans le plan focal ainsi qu'au-dessus et au-dessous de celui-ci, informations « hors foyer » pouvant être mesurées. La comparaison des mesures dans le plan x-y, de préférence en plusieurs positions z, permet l'amélioration de la résolution le long de chacun des axes x, y et z, l'augmentation du nombre de photons utilisés dans le système, et donc l'amélioration du rapport signal-bruit ainsi que la correction des aberrations, telles que les aberrations sphériques ou d'autres aberrations optiques, dans le système optique d'un microscope.
PCT/US2000/011548 1999-04-30 2000-04-26 Procedes et appareil pour une resolution en profondeur amelioree en microscopie, au moyen d'informations hors foyer WO2000067060A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
AU45001/00A AU4500100A (en) 1999-04-30 2000-04-26 Methods and apparatus for improved depth resolution using out-of-focus information in microscopy

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US13198899P 1999-04-30 1999-04-30
US60/131,988 1999-04-30

Publications (1)

Publication Number Publication Date
WO2000067060A1 true WO2000067060A1 (fr) 2000-11-09

Family

ID=22451912

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2000/011548 WO2000067060A1 (fr) 1999-04-30 2000-04-26 Procedes et appareil pour une resolution en profondeur amelioree en microscopie, au moyen d'informations hors foyer

Country Status (2)

Country Link
AU (1) AU4500100A (fr)
WO (1) WO2000067060A1 (fr)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007507744A (ja) * 2003-09-30 2007-03-29 テクニッシェ ユニヴァージテート デルフト 光学像を取得する光学顕微鏡および方法
WO2007104542A1 (fr) * 2006-03-14 2007-09-20 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Procédé et dispositif pour produire une prise de vue fibroscopique non structurée
US8668640B2 (en) 1999-12-17 2014-03-11 Motic China Group Co., Ltd. Methods and apparatus for imaging using a light guide bundle and a spatial light modulator
CN111711750A (zh) * 2020-06-05 2020-09-25 腾讯科技（深圳）有限公司 基于人工智能的图像处理方法、装置、设备及介质

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5760901A (en) * 1997-01-28 1998-06-02 Zetetic Institute Method and apparatus for confocal interference microscopy with background amplitude reduction and compensation
WO1998028655A2 (fr) * 1996-12-20 1998-07-02 Biophysica Technologies, Inc. Instruments optiques confocaux a lumiere modulee et procede correspondant
US5867251A (en) * 1997-05-02 1999-02-02 The General Hospital Corp. Scanning ophthalmoscope with spatial light modulators

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO1998028655A2 (fr) * 1996-12-20 1998-07-02 Biophysica Technologies, Inc. Instruments optiques confocaux a lumiere modulee et procede correspondant
US5760901A (en) * 1997-01-28 1998-06-02 Zetetic Institute Method and apparatus for confocal interference microscopy with background amplitude reduction and compensation
US5867251A (en) * 1997-05-02 1999-02-02 The General Hospital Corp. Scanning ophthalmoscope with spatial light modulators

Non-Patent Citations (8)

* Cited by examiner, † Cited by third party
Title
BARTH M ET AL: "BOOSTING THE OPTICAL TRANSFER FUNCTION WITH A SPATIALLY RESOLVING DETECTOR IN A HIGH NUMERICAL APERTURE CONFOCAL REFLECTION MICROSCOPE", OPTIK,DE,WISSENSCHAFTLICHE VERLAG GMBH.STUTTGART, vol. 96, no. 2, 1 April 1994 (1994-04-01), pages 53 - 58, XP000434220, ISSN: 0030-4026 *
BERTERO ET AL: "Resolution in Diffraction-Limited Imaging, a Singular Value Analysis - IV. The Case of Uncertain Localization or Non-Uniform Illumination of the Subject", OPTICA ACTA., vol. 31, no. 8, 1984, pages 923 - 946, XP000900470, ISSN: 0030-3909 *
BERTERO M ET AL: "Super-Resolution in Confocal Scanning Microscopy", INVERSE PROBLEMS 3, 1987, pages 195 - 212, XP000934212 *
COX I J ET AL: "IMPROVEMENT IN RESOLUTION BY NEARLY CONFOCAL MICROSCOPY", APPLIED OPTICS, vol. 21, no. 5, March 1982 (1982-03-01), NEW YORK US, pages 778 - 781, XP002144725 *
REINHOLZ F ET AL: "A NEW POWERFUL MODE OF LASER SCANNING MICROSCOPY", OPTIK,DE,WISSENSCHAFTLICHE VERLAG GMBH.STUTTGART, vol. 82, no. 4, 1 July 1989 (1989-07-01), pages 165 - 168, XP000048731, ISSN: 0030-4026 *
REINHOLZ F ET AL: "IMAGE ENHANCEMENT BY TRACKING AND SAMPLING IN THE DETECTION PLANE", OPTIK,DE,WISSENSCHAFTLICHE VERLAG GMBH.STUTTGART, vol. 96, no. 2, 1 April 1994 (1994-04-01), pages 59 - 64, XP000434221, ISSN: 0030-4026 *
SHEPPARD C J R ET AL: "IMPROVEMENT IN AXIAL RESOLUTION BY INTERFERENCE CONFOCAL MICROSCOPY", OPTIK,DE,WISSENSCHAFTLICHE VERLAG GMBH.STUTTGART, vol. 87, no. 3, 1 May 1991 (1991-05-01), pages 129 - 132, XP000230044, ISSN: 0030-4026 *
WILSON T ET AL: "EFFECT OF DETECTOR DISPLACEMENT IN CONFOCAL IMAGING SYSTEMS", APPLIED OPTICS,US,OPTICAL SOCIETY OF AMERICA,WASHINGTON, vol. 27, no. 18, 15 September 1988 (1988-09-15), pages 3791 - 3799, XP000000891, ISSN: 0003-6935 *

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8668640B2 (en) 1999-12-17 2014-03-11 Motic China Group Co., Ltd. Methods and apparatus for imaging using a light guide bundle and a spatial light modulator
JP2007507744A (ja) * 2003-09-30 2007-03-29 テクニッシェ ユニヴァージテート デルフト 光学像を取得する光学顕微鏡および方法
WO2007104542A1 (fr) * 2006-03-14 2007-09-20 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Procédé et dispositif pour produire une prise de vue fibroscopique non structurée
US7801405B2 (en) 2006-03-14 2010-09-21 Fraunhofer-Gesellschaft Zur Foerderung Der Angewandten Forschung E.V. Method and device for generating a structure-free fiberscopic picture
CN111711750A (zh) * 2020-06-05 2020-09-25 腾讯科技（深圳）有限公司 基于人工智能的图像处理方法、装置、设备及介质
CN111711750B (zh) * 2020-06-05 2023-11-07 腾讯科技（深圳）有限公司 基于人工智能的图像处理方法、装置、设备及介质

Also Published As

Publication number Publication date
AU4500100A (en) 2000-11-17

Similar Documents

Publication Publication Date Title
US6388809B1 (en) Methods and apparatus for improved depth resolution use of out-of-focus information in microscopy
US6483641B1 (en) Apparatus and methods relating to spatially light modulated microscopy
US7274446B2 (en) Method and arrangement for the deep resolved optical recording of a sample
US8629413B2 (en) Microscopy with adaptive optics
US8730573B2 (en) Adaptive optics microscopy with phase control of beamlets of a light beam
US6888148B2 (en) Arrangement for the optical capture of excited and /or back scattered light beam in a sample
US6825454B2 (en) Automatic focusing device for an optical appliance
US7170696B2 (en) Method and arrangement for deeply resolved optical detection of a sample
US7369309B2 (en) Confocal microscope
US20030142292A1 (en) Arrangement for the optical capture of excited and/or back scattered light beam in a sample
US20030197924A1 (en) Confocal microscope apparatus
GB2321517A (en) Confocal microscopic equipment for measuring solid shapes
JP2002507763A (ja) 広視野領域・高速走査顕微鏡検査法
EP1283416A2 (fr) Lecteur de biopuces et dispositif de formation d'images fluorométriques
US6108127A (en) High resolution confocal microscope
KR101652356B1 (ko) 광학적 웨이퍼 검사 장치
US20100264294A1 (en) Multi-focal spot generator and multi-focal multi-spot scanning microscope
WO2000067060A1 (fr) Procedes et appareil pour une resolution en profondeur amelioree en microscopie, au moyen d'informations hors foyer
Dlugan et al. Improvements to quantitative microscopy through the use of digital micromirror devices

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A1

Designated state(s): AE AL AM AT AU AZ BA BB BG BR BY CA CH CN CU CZ DE DK EE ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MD MG MK MN MW MX NO NZ PL PT RO RU SD SE SG SI SK SL TJ TM TR TT UA UG US UZ VN YU ZA ZW

AL Designated countries for regional patents

Kind code of ref document: A1

Designated state(s): GH GM KE LS MW SD SL SZ TZ UG ZW AM AZ BY KG KZ MD RU TJ TM AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE BF BJ CF CG CI CM GA GN GW ML MR NE SN TD TG

121 Ep: the epo has been informed by wipo that ep was designated in this application
REG Reference to national code

Ref country code: DE

Ref legal event code: 8642

122 Ep: pct application non-entry in european phase
NENP Non-entry into the national phase

Ref country code: JP