WO2021053245A1 - Microscopie de localisation d'éclairage structurée activée par un motif - Google Patents

Microscopie de localisation d'éclairage structurée activée par un motif Download PDF

Info

Publication number
WO2021053245A1
WO2021053245A1 PCT/EP2020/076337 EP2020076337W WO2021053245A1 WO 2021053245 A1 WO2021053245 A1 WO 2021053245A1 EP 2020076337 W EP2020076337 W EP 2020076337W WO 2021053245 A1 WO2021053245 A1 WO 2021053245A1
Authority
WO
WIPO (PCT)
Prior art keywords
activation
pattern
excitation light
patterns
light
Prior art date
Application number
PCT/EP2020/076337
Other languages
English (en)
Inventor
Carlas Sierd SMITH
Johan Piet CNOSSEN
Original Assignee
Technische Universiteit Delft
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Technische Universiteit Delft filed Critical Technische Universiteit Delft
Publication of WO2021053245A1 publication Critical patent/WO2021053245A1/fr

Links

Classifications

    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/06—Means for illuminating specimens
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/62—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light
    • G01N21/63—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light optically excited
    • G01N21/64—Fluorescence; Phosphorescence
    • G01N21/645—Specially adapted constructive features of fluorimeters
    • G01N21/6456—Spatial resolved fluorescence measurements; Imaging
    • G01N21/6458—Fluorescence microscopy
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/0004—Microscopes specially adapted for specific applications
    • G02B21/002—Scanning microscopes
    • G02B21/0024—Confocal scanning microscopes (CSOMs) or confocal "macroscopes"; Accessories which are not restricted to use with CSOMs, e.g. sample holders
    • G02B21/0052—Optical details of the image generation
    • G02B21/0076—Optical details of the image generation arrangements using fluorescence or luminescence
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
    • G02B21/365—Control or image processing arrangements for digital or video microscopes
    • G02B21/367—Control or image processing arrangements for digital or video microscopes providing an output produced by processing a plurality of individual source images, e.g. image tiling, montage, composite images, depth sectioning, image comparison
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/56—Optics using evanescent waves, i.e. inhomogeneous waves
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/58—Optics for apodization or superresolution; Optical synthetic aperture systems

Definitions

  • the disclosure relates to structured illumination and localization microscopy and, in particular, though not exclusively, to methods and systems for structured illumination, and a computer program product enabling a computer system to perform such methods.
  • the sample of interest is labelled with a set of fluorophores, which may be fluorescent molecules or fluorescent labels.
  • the specimen is illuminated with light of a specific wavelength (or wavelengths) which is absorbed by the fluorophores, called excitation light. This causes the fluorophores to emit light of longer wavelengths (i.e. , of a different colour than the absorbed light).
  • excitation light a specific wavelength (or wavelengths) which is absorbed by the fluorophores
  • the illumination light is separated from the much weaker emitted fluorescence through the use of a dichroic mirror and/or emission filter.
  • fluorophores that can be switched between a fluorescent state and a non-fluorescent state by illuminating the fluorophores with activation light, respectively deactivation light.
  • these fluorophores can be stochastically switched between a light emitting state (“on”-state) and a dark state (“off-state) such that at any instant in time only a sparse subset of all fluorophores is in the on-state. This has the consequence that light from individual fluorophores can be measured by a point or array detector such as a camera.
  • the position of the individual emitters in the plane imaged onto the camera can then be determined with a precision, which is typically on the order of 10 nm depending on photon count, about 10-20 times smaller than the diffraction limit. Repeating this process, i.e. recording many time frames of such sparse images and analysing all frames for the emitter locations, finally gives an image of the sample with details on the order of the localization precision rather than on the order of the diffraction limit. Because of the need to collect and analyse many images, the temporal resolution of these methods is relatively low.
  • US10247672 describes a super-resolution method referred to as Pattern Activated Non-Linear Structured Illumination Microscopy (PA-NL-SIM).
  • the method comprises providing patterned activation radiation and patterned (or structured) excitation radiation to a sample that includes photo-transformable optical labels (PTOLs) such as photo-activatable fluorescent molecules, wherein the activation light and the excitation light each have an optical parameter (e.g. intensity) that varies periodically in space, possibly with the same spatial period.
  • PTOLs photo-transformable optical labels
  • SIM structured illumination microscopy exploits the fact that when two patterns are superimposed multiplicatively a beat pattern (similar to a Moire pattern) will appear in the product of the two patterns.
  • One pattern can be the unknown sample structure emitting signal light, and the other pattern can be the purposely structured pattern of excitation light. Because the amount of signal light emitted from a point in the sample is proportional to the product of the local excitation light intensity and the relevant structure of the sample, the observed signal light image that is detected by the detector will show the beat pattern of the overlap of the two underlying patterns. Because the beat pattern can be coarser than those of the underlying patterns, and because the illumination pattern is known, the information in the beat pattern can be used to determine the normally unresolvable high-resolution information about the sample.
  • PA-NL-SIM uses a combination of an activation pattern and a excitation pattern to generate a fluorescence emission pattern comprising additional information.
  • WO201 7/153430 describes a method for high-resolution imaging of a structure with fluorescent markers, comprising scanning (part of) a sample with a zero point of an illumination intensity distribution, where the zero point is typically surrounded by a ring-shaped or doughnut-shaped intensity maximum.
  • An activation region comprising and surrounding the zero point may be irradiated with activation radiation, and the activation region may be surrounded by and partially overlap with a deactivation region.
  • switchable luminescence markers the whole sample may be scanned by varying the active regions over a grid and within the active regions, varying the scanning zero point.
  • MIN FLUX uses (or can be regarded as an extension of) the MIN FLUX method, described in Balzarotti et al., ‘Nanometer resolution imaging and tracking of fluorescent molecules with minimal photon fluxes’, Science Vol. 355, Iss. 6325, pages 606-612.
  • MINFLUX uses the same excitation light zero-point scanning as the method disclosed in WO201 7/153430, but no activation light or deactivation light.
  • MINFLUX nanoscopy delivers multicolor nanometer 3D-resolution in (living) cells’, available at https://www.biorxiv.Org/content/10.1101 /734251 v1.
  • SIMFLUX may be considered a combination of MINFLUX with Structured Illumination Microscopy (SIM).
  • SIMFLUX The structured illumination in SIMFLUX consists of quickly switched (with respect to position and orientation) sinusoidal excitation light patterns to illuminate a sample including fluorescent molecules.
  • the fluorescent photon count is recorded and stored for a number of frames, the number depending on the number of excitation patterns.
  • the information from the frames is combined to determine a molecule position, based on the diffraction pattern, the intensity, and the excitation pattern.
  • the combination of the excitation patterns provides a uniform illumination of the sample leading to a uniform increase in localization precision.
  • This technique results in an image resolution gain of 2.2 or an reduction of the required illumination light by a factor of 4.84 when compared with state of the art methods that use uniform illumination.
  • the invention may relate to a method for structured illumination microscopy comprising: illuminating an exposure area of a sample comprising photoactivatable molecules with a predetermined, inhomogeneous activation light pattern comprising one or more activation light maxima, the activation light activating one or more of the photoactivatable fluorescent molecules; sequentially illuminating the exposure area with a plurality of excitation light patterns, the plurality of excitation light patterns defining a cumulative excitation light pattern, the cumulative excitation light pattern defining one or more cumulative excitation light minima, the cumulative excitation light pattern being positioned relative to the activation pattern, such that each maximum of the activation light pattern coincides with one of the one or more cumulative excitation light minima, the excitation light of an excitation light pattern exciting one or more of the activated fluorescent molecules; detecting radiation emitted by the photoactivatable fluorescent molecules which are activated by the activation light pattern
  • the position information may be used to e.g. reconstruct an image, preferably a super-resolution image, of the at least part of the sample.
  • the determining of the position information may use information on the activation light pattern and on the plurality of excitation light patterns.
  • An activation light pattern may also be referred to as an activation pattern, and an excitation light pattern may be referred to as an excitation pattern.
  • the structured illumination microscopy scheme uses illumination patterns which are selected such that a cumulative illumination intensity of a series of illumination patterns is inhomogeneous, comprising low intensity areas (minima) and high intensity areas (maxima). That way, the illumination patterns may be selected to maximize the information per photon, in particular the position information.
  • the inhomogeneous cumulative excitation light pattern is combined with an inhomogeneous activation light pattern, comprising low intensity areas (minima) where there is a low probability of activating a fluorescent molecule, and high intensity areas (maxima) where there is a high probability of activating a fluorescent molecule.
  • an inhomogeneous activation light pattern comprising low intensity areas (minima) where there is a low probability of activating a fluorescent molecule, and high intensity areas (maxima) where there is a high probability of activating a fluorescent molecule.
  • Fluorescent molecules generally have a ‘fixed’ photon budget, typically about 1500-5000 photons per molecule (such as e.g. Alexa Fluor 647), before they stop functioning. Therefore, maximising the information per photon not only reduces the sampling time to obtain a given localisation precision, but also increases the maximally obtainable localisation precision.
  • the information per photon can effectively be increased by adding a priori information, e.g. by using a known activation and/or excitation pattern, which may result in a statistical probability distribution of localisations.
  • This process may also be executed in closed loop: once initial information on the position of molecules has been obtained, the activation and/or excitation light patterns may be re-optimized to further increase the information per photon, and hence the localisation precision. This process of re optimisation may be repeated for a number of frames.
  • the information per photon may be the Fischer information per photon emitted by the photoactivatable fluorescent molecules, where the Fischer information may relate to a position of a fluorescent molecule.
  • the position may be estimated using a maximum likelihood estimation algorithm.
  • the method may comprise determining maxima of the Fischer information divided by the cumulative excitation pattern, based on at least the plurality of excitation patterns.
  • the method may further comprise positioning the plurality of excitation patterns relative to the activation pattern maxima such that the maxima of the Fischer information divided by the cumulative excitation pattern coincide with the maxima of the activation pattern. This way, the information per photon may be maximised, and consequently, the total available information before the fluorescent molecules go dark.
  • maxima of the Fischer information per photon may coincide with minima in the cumulative excitation pattern.
  • positioning the plurality of excitation patterns such that the one or more maxima of the Fischer information divided by the cumulative excitation light pattern coincide with the one or more maxima of the activation light pattern may comprise positioning the plurality of excitation light patterns such that one or more minima of the cumulative excitation light pattern coincides with the one or more maxima of the activation light pattern.
  • a minimum in the cumulative excitation light may be obtained by using for example different excitation patterns with coinciding minima for at least some of the minima, or, preferably, by using spatially shifted versions of the same excitation pattern, wherein the shift is smaller than the pattern size.
  • different excitation patterns with coinciding minima for at least some of the minima, or, preferably, by using spatially shifted versions of the same excitation pattern, wherein the shift is smaller than the pattern size.
  • many other patterns are also possible, including even random patterns, provided the pattern is known with sufficient precision and comprises intensity gradients and preferably intensity zero points.
  • the information per photon is highest close to the minima (or ideally: zero-points) of the excitation illumination patterns.
  • the method may further comprise: determining a scanning pattern defining scanning positions, and, repositioning the activation light pattern and the plurality of excitation light patterns relative to the sample in accordance with the scanning pattern and, for each scanning pattern position, repeating the previously defined method steps, i.e.
  • the number of scanning positions may depend on the distance between the high activation light intensity areas relative to the size of the high activation light intensity areas. In some embodiments, 2x2 scanning positions may be sufficient, other embodiments may use more or less.
  • the plurality of excitation light patterns may comprise a first series of sinusoidal light patterns with a first orientation, wherein the sinusoidal light patterns have the same period and have equidistant phases, the maximum phase difference being in the interval 30 to 150 degrees, preferably in the interval 60 to 90 degrees.
  • a sinusoidal excitation pattern can be advantageously created by interference of two plane waves. Additional excitation patterns may advantageously be created by phase shifting the interference pattern.
  • the plurality of excitation light patterns may comprise two or more series of sinusoidal light patterns having the same orientation, wherein the orientations of the series may be equally distributed over 180 degrees.
  • a plurality of excitation patterns may comprise two orthogonal series of sinusoidal excitation patterns, or three series of sinusoidal excitation patterns with a mutual angle of 60°.
  • the activation light pattern and/or the plurality of excitation light patterns may be periodic pattern in one or more directions.
  • the activation light pattern may define a regular rectangular, preferably square, grid with a grid distance
  • the plurality of excitation light patterns may comprise a first series of sinusoidal light patterns in a first orientation and a second series of sinusoidal wave patterns in a second orientation, the second series being orthogonal to the first series, and the excitation light patterns having a period equal to the grid distance.
  • the activation light pattern may define a regular triangular grid with a grid distance
  • the plurality of excitation light patterns may comprise a first series of sinusoidal wave patterns in a first orientation, a second series of sinusoidal wave patterns in a second orientation, and a third series of sinusoidal wave patterns in a third orientation, the angle between the first and second orientations being 60 degrees and the angle between the second and third orientations being 60 degrees, and the excitation light patterns having a period equal to the grid distance.
  • Such an embodiment may be advantageous when e.g. an existing, conventional SIM-setup using three pattern orientations is adapted to make use of elements of this disclosure.
  • each of the series of sinusoidal wave patterns may comprise three sinusoidal wave patterns.
  • Three excitation light patterns per series of excitation light patterns may be a suitable number to strike a balance between increase in localisation precision, exposure time and cumulative exposure luminosity. This may result in, on average, a high information to photon count ratio.
  • the method may further comprise: detecting a background and selecting a plurality of excitation patterns based on the detected background. At least the optimal phase distance between sinusoidal waves in a series of excitation patterns may depend on the detected background.
  • the method may further comprise: determining new activation light patterns and/or excitation light patterns, based on the determined position information of one or more of the photoactivatable fluorescent molecules; and, executing the method steps of one or more of the methods described above using the new activation light patterns and/or excitation light patterns.
  • the activation light patterns and/or excitation patterns may be (re-)optimised to further increase the information per photon.
  • An example of such an optimisation is a (phase) shift of excitation patterns relative to an activation pattern.
  • the invention in a second aspect, relates to a microscope for obtaining super-resolution images of a sample comprising photoactivatable fluorescent molecules, the microscope comprising: a sample holder for holding the sample; a first light source and first beam forming optics configured for illuminating an exposure area of the sample with activation light having a predetermined, inhomogeneous activation light pattern, the activation light pattern defining activation light maxima; a second light source and second beam forming optics configured for illuminating the exposure area with a plurality of excitation light patterns, the plurality of excitation light patterns defining a cumulative excitation light pattern, the cumulative excitation light pattern defining one or more cumulative excitation light minima, the plurality of excitation light patterns being positioned relative to the activation pattern, such that each maximum of the activation light pattern coincides with one of the one or more cumulative excitation light minima, the excitation light of an excitation light pattern exciting one or more of the activated fluorescent molecules; a detector for detecting radiation emitted by the photoactiv
  • the processor may use the position information for generating an image of at least part of the sample on the basis of the detected radiation.
  • the second beam forming optics may comprise a computer-controllable polarisation controlling element and one or more gratings, configured to generate a plurality of pre-determined interference patterns.
  • the invention may relate to an optical module for use in a localisation microscope, the optical module being configured for determining position information of one or more photoactivatable fluorescent molecules in a sample, the sample defining a sample plane, the optical module further being configured for: illuminating an exposure area of a sample with a predetermined, inhomogeneous activation light pattern, the activation pattern defining one or more activation light maxima and one or more activation light minima in the sample plane, the activation light activating one or more of the photoactivatable fluorescent molecules; sequentially illuminating the exposure area with a plurality of excitation light patterns, the plurality of excitation light patterns defining a cumulative excitation light pattern, the cumulative excitation light pattern defining one or more cumulative excitation light minima in the sample plane, the plurality of excitation light patterns being positioned relative to the activation light pattern such that each activation light maxima coincides with one of the one or more cumulative excitation light minima, the excitation light of an excitation light pattern exciting one or more of
  • the optical module may further be configured for illuminating the exposure area with one or more excitation light patterns in a directional orthogonal to the sample plane.
  • a module may comprise a second light source for illuminating an exposure area of a sample with excitation light and beam-forming optics for generating an excitation pattern in the sample in the axial direction.
  • beam-forming optics may comprise means for generating a self interference pattern in the excitation light in the axial direction, orthogonal to the sample plane.
  • the invention may relate to a structured light microscopy method comprising: illuminating a sample comprising photoactivatable fluorescent molecules with an axial activation light pattern, the axial activation pattern comprising one or more activation light maxima in an axial direction of the sample, the activation light activating one or more of the photoactivatable fluorescent molecules; sequentially illuminating the sample with a plurality of axial excitation light patterns, the plurality of axial excitation light patterns defining a cumulative excitation light pattern, the cumulative excitation light pattern defining one or more cumulative excitation light minima in the axial direction of the sample, the plurality of cumulative excitation light patterns being positioned relative to the activation pattern such that each maximum of the activation light pattern coincides with one of the one or more cumulative excitation light minima, the excitation light of an axial excitation light pattern exciting one or more of the activated fluorescent molecules; detecting radiation emitted by the photoactivatable fluorescent molecules which are activated by the activation light pattern and excited the plurality of axial excitation excitation
  • the invention may relate to a microscope for obtaining super-resolution images of a sample comprising photoactivatable fluorescent molecules, the microscope comprising: a sample holder for holding the sample; a first light source and first beam forming optics configured for illuminating the sample with activation light having an axial activation light pattern, the axial activation light pattern defining one or more activation light maxima in an axial direction of the sample; a second light source and second beam forming optics configured for illuminating the sample with a plurality of axial excitation light patterns, the plurality of axial excitation light patterns defining a cumulative excitation light pattern, the cumulative excitation light pattern defining one or more cumulative excitation light minima in the axial direction of the sample, the cumulative excitation light pattern being positioned relative to the axial activation pattern, such that each maximum of the axial activation light pattern coincides with one of the one or more cumulative excitation light minima, the excitation light of an axial excitation light pattern exciting one or more of the activated fluorescent molecules;
  • the second beam forming optics may be configured for illuminating the sample from a first direction with a first beam of plane waves of coherent excitation light, and for illuminating the sample from a second direction, the second direction being the counter-direction of the first direction, with a second beam of plane waves of coherent excitation light, the first beam and the second beam interfering to generate an axial excitation light pattern.
  • the invention may relate to a computer comprising a computer readable storage medium having computer readable program code embodied therewith, and a processor, preferably a microprocessor, coupled to the computer readable storage medium, wherein responsive to executing the computer readable program code, the processor is configured to perform the method steps according to any of the process steps described above.
  • the invention may also relate to a computer program product comprising software code portions configured for, when run in the memory of a computer, executing the method steps according to any of the process steps described above.
  • the invention may also relate to a non-transitory computer-readable storage medium storing at least one software code portion, the software code portion, when executed or processed by a computer, is configured to perform, executing the method steps according to any of the process steps described above.
  • aspects of the present invention may be embodied as a system, method or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit," “module” or “system”. Functions described in this disclosure may be implemented as an algorithm executed by a microprocessor of a computer. Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied, e.g., stored, thereon.
  • the computer readable medium may be a computer readable signal medium or a computer readable storage medium.
  • a computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.
  • a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
  • a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof.
  • a computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
  • Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber, cable, RF, etc., or any suitable combination of the foregoing.
  • Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including a functional or an object oriented programming language such as Java(TM), Scala, C++, Python or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages.
  • the program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer, server or virtualized server.
  • the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
  • LAN local area network
  • WAN wide area network
  • Internet Service Provider an Internet Service Provider
  • These computer program instructions may be provided to a processor, in particular a microprocessor or central processing unit (CPU), or graphics processing unit (GPU), of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer, other programmable data processing apparatus, or other devices create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
  • a processor in particular a microprocessor or central processing unit (CPU), or graphics processing unit (GPU), of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer, other programmable data processing apparatus, or other devices create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
  • CPU central processing unit
  • GPU graphics processing unit
  • FPGAs Field-Programmable Gate Arrays
  • ASICs Application-specific Integrated Circuits
  • ASSPs Application- Specific Standard Products
  • SOCs System-on-a-Chip systems
  • CPLDs Complex Programmable Logic Devices
  • These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
  • the computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
  • each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s).
  • the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.
  • Fig. 1A and 1B illustrate a known structured illumination microscopy method
  • Fig. 2 depicts a schematic of the formation of an activation pattern for a structured illumination method according to an embodiment of the invention
  • Fig. 3A and 3B illustrate activation patterns for a structured illumination method according to an embodiment of the invention
  • Fig. 4 illustrates a schematic exposing a sample to an activation pattern for a structured illumination according to an embodiment of the invention
  • Fig. 5 depicts a block diagram of a patterned activation structured illumination microscopy method according to an embodiment of the invention
  • Fig. 6 schematically depicts a system for a pattern-activated structured illumination method according to an embodiment of the invention
  • Fig. 7A and 7B schematically illustrate the gain in localization accuracy that may be achieved using the embodiments described in this application;
  • Fig. 8 illustrates a patterned activation structured illumination microscopy method according to an embodiment of the invention
  • Fig. 9 schematically depicts part of a structured illumination microscopy system according to an embodiment of the invention.
  • Fig. 10A-10C depict the localization improvement in the axial direction for a pattern-activation structured illumination microscopy scheme according to an embodiment of the invention.
  • Fig. 11 is a block diagram illustrating an exemplary data processing system that may be used for executing methods and software products described in this application.
  • FIG. 1A depicts a known structured illumination microscopy method.
  • the figure illustrates a schematic on the working of the so-called SIMFLUX method as described by Cnossen et al. in their article ‘Localization microscopy at double precision with patterned illumination’] only a single orientation is shown.
  • a sample comprising a fluorescent molecule 102 is illuminated with excitation light with a series of three sinusoidal wave intensity patterns 104 I-3 with 3 phase shifted patterns with equal phase steps of 120° per orthogonal orientation of the line pattern.
  • the excitation pattern has high intensity areas 108 (dark) and low intensity areas 110 (light).
  • the fluorescent radiation is detected and an image is recorded, resulting in three images 114 I-3 for a first orientation of the line pattern.
  • Each image comprises pixels, where the pixel value represents the photon count. A higher photon count is represented with a lighter colour.
  • First localization information may be obtained by estimating the centroid of the bright spots, i.e. the regions with a photon count higher than the background noise. The estimated centroid location is indicated by a circle 112 I-3 , with the size of the circle proportional to the uncertainty in centroid location.
  • Second localization information may be obtained by correlating the photon count 120 I-3 with the excitation pattern 122 as shown in graph 124.
  • a high photon count is associated with a high excitation light intensity, and a low photon count with a low excitation light intensity.
  • the photon counts for the three images per orientation are known to be spaced (in this case) 120° apart. As the position of the excitation pattern is known, this provides additional a priori information (modulo the wavelength of the excitation pattern).
  • the first localization information and the second localization information may be combined to effectively increase the localization precision.
  • the same procedure may be executed with a second series of excitation patterns orthogonal to the first series of excitation patterns, to increase the localization precision in both directions. The measurement may be repeated several times.
  • the measurements may then be combined 126 into a localization image 128, in which the estimated location 130 of the fluorescent molecule is shown with an “X” and a circle representing the uncertainty.
  • This method results in an improvement of localization precision with a factor of about 2.2, compared to conventional position estimation with uniform illumination, i.e. Single Molecule Localisation Microscopy (SMLM).
  • SMLM Single Molecule Localisation Microscopy
  • Fig. 1B depicts a one-dimensional cross-section of illumination intensities of a structured illumination microscopy method as described with reference to Fig. 1 A. It is not desired to have the localization precision vary over the field of view (FoV), therefore a substantially uniform localization precision is obtained by phase-shifting sinusoidal excitation patterns with equal phase shifts as shown in the figure.
  • a series of excitations patterns 132 comprises three sinusoidal 120 degree phase-shifted excitation patterns 134 I-3 , that correspond to the patterns 104-i_ 3 in Fig. 1.
  • the local minima 136-i_ 3 of the excitation patterns are equally spaced over a sample extending along the horizontal axis.
  • the total received excitation light 138 is the same for each point in the sample.
  • the average improvement drops if non-equal phase steps are used because the improvement varies non-uniform ly over the FoV.
  • a non-uniform intensity distribution of the excitation light in the FoV may be used to increase the performance of a structured light microscopy scheme, such as the SIMFLUX method.
  • the invention exploits characteristics of certain excitation light patterns that have a non-uniform cumulative intensity distribution, wherein in some areas of the excitation light pattern the improvement in localization drops compared to known structured microscopy methods (such as the SIMFLUX scheme), while in other areas of the excitation light pattern the improvement in localization increases compared to known structured microscopy methods.
  • predetermined activation light patterns may be used to increase the performance of the structured light microscopy scheme.
  • the activation patterns will stochastically activate photo- activatable fluorescent proteins at targeted positions over the FoV. This creates an additional source of a priori information for improving the localization of the fluorophores, e.g. fluorescent proteins.
  • the additional a priori knowledge regarding the position of fluorophores in the FoV effectively increases the amount of information per photon. Combining the use of the activation and excitation light patterns will provide a substantial improvement in the performance compared to state of the art structured light microscopy schemes.
  • Fig. 2 depicts one-dimensional cross-section of an example of activation and excitation light patterns according to an embodiment of the invention.
  • a sample comprising photoactivatable fluorescent molecules and defining a sample plane may be illuminated with activation light, wherein the activation light has a wavelength configured to activate photoactivatable fluorescent molecules in the sample.
  • the activation light 202 has a spatial intensity pattern defining one or more activation areas 204, wherein an activation area in the sample plane may define an area with a high activation light intensity surrounded by regions with low activation light intensity.
  • An activation area may e.g. be defined as the FWHM region or twice the FWHM region of a local maximum in an activation light pattern, or as e.g. the region with a >50 % or >95% activation probability.
  • the sample may be exposed to a plurality of activation areas wherein each activation area may be shaped according to the spatial intensity pattern of Fig. 2.
  • the plurality of activation areas may be arranged in a 2-dimensional plane forming an activation pattern.
  • the activation light intensity is sufficient to activate photoactivatable fluorescent molecules.
  • the probability that a photoactivatable fluorescent molecule is activated is proportional to the intensity of the activation light at the position of the photoactivatable fluorescent molecule. Due to the shape of the spatial intensity pattern, this probability is negligible for the area outside the activation areas.
  • the activation light pattern may have any sort of structure, and may for instance be structured as pin points in a regular square grid or triangular grid.
  • Activation light patterns may be regular or irregular; examples of regular patterns are the aforementioned square or triangular grids or sinusoidal waves; in principle, any pattern can be used as activation pattern or excitation pattern in a series of excitation patterns, provided the activation pattern can be configured to activate primarily or exclusively photo-activatable molecules in the high-precision regions of the series of excitation patterns, in particular photo-activatable molecules in the regions with a relatively low cumulative excitation light dose defined by the series of excitation patterns.
  • Activation light patterns may also be formed by light sheets.
  • An advantage of using engineered light sheets is that activation may be limited to a single layer or small region in the axial direction, thus reducing background noise and reducing phototoxicity.
  • One or more parallel light sheets may be engineered for providing activation patterns and/or excitation patterns in the sample plane with a limited axial extent.
  • the sample After illumination with the activation light, the sample may be illuminated with a consecutive series of excitation light patterns 206.
  • the sample may be illuminated with a consecutive series of excitation light patterns 206.
  • the series may comprise a plurality (e.g. three) of excitation light patterns having a sinusoidal shape along an axis of the image plane (defining one orientation in the image plane).
  • excitation light patterns may be used in two orthogonal orientations or three 60° spaced orientations. Other arrangements however are also possible.
  • the sample may be consecutively illuminated with the excitation patterns, i.e. the excitation patterns are used one after another.
  • the series may alternate between the plurality of orientations.
  • the sample may be illuminated using a first sinusoidal excitation light pattern in a first orientation. Thereafter, the sample may be sequentially illuminated by the first sinusoidal excitation light pattern in a second orientation, a second sinusoidal excitation light pattern in the first orientation, the second sinusoidal excitation light pattern in the second orientation, etc.
  • the second sinusoidal excitation light pattern may be a phase-shifted version of the first sinusoidal excitation light pattern.
  • the excitation light may excite one or more of the activated fluorescent molecules with a probability that is proportional to the intensity of the excitation light.
  • the excitation light pattern may have low-intensity regions, typically local minima 208 I-3 and high-intensity regions.
  • the excitation light pattern and the activation light pattern are configured such that low intensity regions of the excitation patterns 208 I-3 are located in or near the activation areas 204.
  • An activation area may be defined as the area where the intensity of the activation light is greater than or equal a predetermined percentage of a maximum activation light intensity, e.g. at least 50 % or at least 25 % of the local maximum activation light intensity.
  • An activation area may also be defined as the area where the probability of activating a light-activatable molecule is at least a predetermined number, e.g.
  • the excitation patterns may have the same period as the activation pattern, or an integer multiple thereof.
  • each minimum in the excitation patterns is associated with an activation area, to minimize phototoxicity.
  • the cumulative excitation light 210 exhibits a pattern with local minima and local maxima, wherein the local minima approximately coincide with maxima of the activation pattern. This way, the amount of information per photon is optimised.
  • the excitation light patterns may have any sort of structure, preferably selected depending on the activation pattern used.
  • Excitation light patterns may be regular or irregular; examples of regular patterns are the sinusoidal waves shown in Fig. 2, or square or triangular grids. If use is made of an irregular activation pattern, the excitation pattern may be similarly irregular, the excitation patterns and/or the cumulative excitation pattern preferably having minima at or near the maxima of the activation light pattern, as this way, the high information per photon regions of the activation pattern coincide with the high information per photon regions of the series of excitation patterns.
  • excitation light patterns may be formed by light sheets, providing similar advantages, e.g., limiting excitation to a single layer or small region in the axial direction, thus reducing background noise and reducing phototoxicity.
  • Activation light sheets and excitation light sheets may be used in combination or separately.
  • excitation light patterns in a consecutive series of excitation light patterns of the same orientation may be phase-shifted copies of each other.
  • an excitation light pattern may be reused by a pattern generating device.
  • the increase in localization precision increases with the frequency of the excitation light pattern. Therefore, in an embodiment, the maximum possible frequency for each excitation pattern may be used.
  • the phase step (phase difference) between such phase-shifted copies may be selected based on other parameters, e.g. the influence of the background. In general, the phase step should strike a balance between a large enough phase step that different excitation patterns lead to a detectably different response, and a small enough phase step that a sufficient increase in localization precision is obtained.
  • the activation light pattern and/or the excitation light patterns may be adjusted, e.g. shifted, to maximise the information per photon. For example, after illumination with an activation light pattern and a first cycle of excitation light patterns, information from the detected photons may be used to adjust the position of the excitation patterns in a second cycle.
  • the excitation light patterns may e.g. all be shifted with the same amount, or the phase difference between excitation light patterns may be adjusted. For example, if there are many detections close to activation light maxima, the phase difference between excitation light patterns may be reduced, lowering the cumulative excitation radiation near activation light maxima.
  • the radiation emitted by the activated and excited fluorescent molecules may be detected by (an array of) point detectors, such as a camera, and processed to obtain a localization information improvement, which may in turn be used to reconstruct an image, preferably a localization image, of at least part of the sample.
  • an array of point detectors such as a camera
  • the excitation light patterns may be selected such that a cumulative or combined, e.g. summed or time-integrated, excitation intensity 210 of a series of excitation patterns 210 is inhomogeneous, comprising low intensity areas and high intensity areas.
  • a minimum in the cumulative excitation light may be obtained by e.g. using different excitation light patterns with coinciding minima for at least some of the minima such as a sinusoidal light pattern with (odd) integer multiple periods of a certain base frequency, or, preferably, by using spatially shifted versions of the same excitation pattern, wherein the shift is smaller than the pattern size.
  • the aforementioned localization precision enhancing properties of the combined activation and excitation light patterns are strongest close to the minima (or ideally: zero-points) of the excitation light patterns, as the localization precision enhancement, or increase in information per photon, relative to standard Single- Molecule Localization Microscopy depends on the ratio between the gradient of the intensity of the excitation pattern and the intensity of the excitation pattern itself.
  • excitation patterns which are selected such that the cumulative excitation pattern is non-uniform, therefore results in high-resolution areas, corresponding to the minima of the cumulative excitation pattern, and low-resolution areas, corresponding to the maxima of the cumulative excitation pattern.
  • Activating the fluorescent molecules around the minima of the excitation illumination patterns will result in the selection of areas with a high information content per photon. Moreover, as the location of the activation area is known, additional a priori information is added which may be used to increase the accuracy of the localization estimate. By shifting the activation and excitation patterns relative to the sample, the entire sample, or at least a relevant portion of it, may be scanned.
  • the use of an activation pattern has two additional benefits besides a localisation improvement. The fluorescent molecules are more likely to be activated in or near areas of optimum localisation precision.
  • this region may be of the order of 0.1 micron to 5 micron, preferably about 0.5 to 1.5 micron, for instance about 850 nm; this way the overall exposure of the fluorescent molecules is reduced and bleaching effects can be suppressed.
  • the background radiation (“noise”) may be reduced by limiting the number out of focus of fluorescent molecules that contribute to the background.
  • the aforementioned information per photon i.e. per photon emitted by a fluorescent molecule, may be referred to as the Fischer information, which is a well- established concept to define how much we know about a specific estimated variable.
  • the photons may be detected by a camera with, typically, a two- dimensional array of detector elements corresponding to pixels in an image obtained by the camera.
  • the detection of photons, and hence the intensity values in the image are generally distributed according to a Poisson distribution, also known as shot- noise.
  • a fluorescent molecule gives only a limited number of photons before going into a dark state, and as a result, it is advantageous to maximize the amount of information per photon.
  • an aim is to determine spatial coordinates (x, y, z) of the molecule in 2 or 3 dimensions by determining spatial information ( q c , q n and, optionally, q z ) based on the detected photons, e.g. using a maximum likelihood estimation algorithm.
  • Determination of the Fisher information is generally based on a model /1 ⁇ 2 , /(#) representing the expected value of a pixel k in a camera frame / for a given set of molecule parameters Q, e.g. location parameters 0 x y z , fluorescent molecule photon count parameters q,, and parameters for a model of the fluorescence background b k (9 b ).
  • the model m 3 ⁇ 4 (0) may be defined as: where b k (9 b ) is the number of background photons, PSF fc (0) is the point spread function of the optical system, A k is the area of pixel k and P f (9) defines the excitation light intensity at frame /.
  • the expected number of detected photons from the fluorescent molecule is proportional to the excitation light intensity.
  • the Fischer information matrix 1(0) of the set of parameters Q for a measurement over F frames, each frame comprising N pixels, may then be given by: where I(0) i; - is the (i,j) element of the matrix. That is, the Fischer information depends on a sum over all pixels in all frames of the relative derivative of the number of detected photons and is generally highest where a high gradient is combined with a low value of p kj ⁇ 9).
  • m 3 ⁇ 4 (0) may depend on the variables that are estimated, which may include e.g. location information and a photon count 9 h and other aspects of the measurements, such as fluorescent background radiation and properties of the point spread function.
  • the method may comprise determining maxima of the Fischer information divided by the cumulative excitation pattern, based on at least the plurality of excitation patterns.
  • the method may further comprise positioning the plurality of excitation patterns relative to the activation pattern maxima such that the maxima of the Fischer information divided by the cumulative excitation pattern coincide with the maxima of the activation pattern. This way, the information per photon may be maximised, and consequently, the total available information before the fluorescent molecules go dark.
  • the combination of excitation patterns may be chosen such that is maximized at or close to a maximum of the activation pattern, is the expected number of photons from the emitting fluorescent molecule.
  • a trade-off can be made between the estimation precision of the parameters.
  • maxima of the Fischer information per photon may coincide with minima in the cumulative excitation pattern 210.
  • positioning the plurality of excitation patterns 208 I-3 such that the one or more maxima of the Fischer information divided by the cumulative excitation light pattern coincide with the one or more maxima of the activation light pattern may comprise positioning the plurality of excitation light patterns such that one or more minima of the cumulative excitation light pattern coincides with the one or more maxima of the activation light pattern.
  • Fig. 3A and 3B depict examples of two-dimensional scanning patterns according to various embodiments of the invention.
  • Scanning patterns 302,304 may be used to scan a part of a sample with an activation pattern and corresponding excitation patterns.
  • Other embodiments may use different scanning patterns, preferably scanning patterns that are periodic in one or more directions.
  • a first activation pattern 302 covering part of a sample may comprise activation areas 306-i_ n , depicted as solid spots.
  • a sinusoidal light pattern along the main directions of the grid 316,318 may be used to excite fluorescent molecules in the sample.
  • the sinusoidal waves may be selected to have the same period as the activation pattern.
  • the distance between two minima of an excitation pattern may be substantially equal to the distance between to neighbouring activation areas.
  • the position of the activation areas relative to the position of the minima of the sinusoidal light patterns may be known with great precision, e.g. better than 1 nm.
  • the sample After illuminating the sample with a first activation pattern, e.g. the shown activation pattern, the sample may be illuminated with excitation light in a series of excitation patterns, and measurements associated with the first activation pattern may be obtained. Sometime after the illumination with the activation pattern, most activated fluorescent molecules may have become deactivated.
  • the sample may then be illuminated with a shifted activation pattern, e.g. shifted along a first principal axis 318, such that e.g. activation area 306 n is shifted to a new position 308. Localization measurements associated with the shifted activation pattern may then be obtained.
  • the same procedure may be repeated, shifting the activation pattern along the principal axes 316,318 such that activation area 306 n is shifted to additional positions 310,312, for which additional localization measurements may be obtained.
  • the diameter of the activation area may be approximately half the period of the activation pattern.
  • the diameter of the activation area may be smaller than half the period of the activation pattern. In such an embodiment, more scanning positions may be required. Reducing the size of the activation area may require longer measuring times, but may provide a higher localization precision or less interference from nearby activation areas.
  • Fig. 3B illustrates a second activation pattern 304 comprising activation areas depicted as solid circles. Similar as explained above, the activation pattern may be shifted to effectively scan a sample; e.g. activation area 320 may be shifted along axes 326,328,330 to positions 322,324, thus covering most of the sample. Measurements may be obtained by illuminating the sample with e.g. sinusoidal excitation patterns in the directions of the axes 326,328,330 for each position of the activation pattern.
  • An advantage of a triangular pattern 304, compared to a square pattern 302 may be that less activation pattern positions are required to cover the sample, and that the method may be more easily combined with e.g. a structured illumination microscope generating sinusoidal light patterns in three directions. Different embodiments may use different activation patterns.
  • Fig. 4 depicts a schematic illustration of a plurality of time instances -A , in this case four time instances, of a single region 450 of a sample, wherein at each time instance a particular part of the region is exposed to activation and excitation light.
  • the depicted region may have a size of about 220 nm by 220 nm.
  • the sample may be illuminated with activation light defining an activation area 452.
  • the sample may be illuminated with activation light defining an activation area 460, which corresponds to area 454 in the depiction of the first time instance.
  • the activation pattern may be shifted to create an activation area in position 462, corresponding to area 456, and in a fourth time instance, the activation area may be shifted to position 464, corresponding to area 458.
  • the positions may be probed in a different order.
  • the positions 452,454,456,458 may also be referred to as first, second, third and fourth scanning position.
  • the sample may be illuminated with a series of excitation patterns.
  • Cross-sections of an activation light pattern 472 and excitation light patterns 470-i_ 6 are schematically shown for a horizontal direction 466 and for a vertical direction 468.
  • the cross-sections 472 of the activation light pattern may show (for each time instance) a single activation pattern.
  • the excitation light patterns 470 I-6 only one dimension of each cross-section is shown, the cross-section in the other direction may be essentially constant.
  • Excitation light corresponding to the six excitation patterns may be used to illuminate the sample, one pattern after another.
  • the excitation patterns corresponding to one activation area may be used in any order.
  • the four time instances may form an example measurement sequence, where the activation pattern and excitation patterns may be moved around the field of view together to measure all molecules with optimal localization precision.
  • Fig. 5 depicts a block diagram of a structured illumination microscopy method according to an embodiment of the invention.
  • a sample may be provided 502, preferably to a sample holder of a fluorescence microscope.
  • the sample may comprise photoactivatable fluorescent molecules.
  • At least part of the sample may be illuminated 504 with activation light that has a spatial activation pattern.
  • the activation pattern may define one or more activation areas, preferably a plurality of activation areas in a grid such as a square or triangular grid. In the activation areas, the intensity of the activation light is sufficient to activate one or more photoactivatable fluorescent molecules.
  • the sample is subsequently illuminated with excitation light in a consecutive series of spatial excitation patterns.
  • the excitation patterns may be selected based on the activation pattern.
  • the excitation patterns may have, in one or more orientations, the same spatial period as the activation pattern.
  • the activation pattern may form a square grid
  • the series of excitation patterns may comprise two orthogonal partial series of light patterns, as explained in more detail with reference to Fig. 4.
  • a partial series of excitation light patterns having the same orientation may comprise phase shifted copies of a single excitation light pattern.
  • the excitation light patterns may be selected such that a cumulative or combined excitation pattern, e.g. a sum or time integral, may have intensity minima that at least approximately coincide with the maxima of the activation pattern.
  • the excitation patterns may be chosen such that a minimum of each excitation pattern overlaps with an activation area.
  • the excitation light may excite one or more of the activated fluorescent molecules.
  • the different excitation light patterns may be generated one after another. At least a full series of excitation light patterns may expose part of the sample during an activation time of the photoactivatable fluorescent molecules.
  • the fluorescent radiation emitted by the activated and excited fluorescent molecules may be detected 508 by a photodetector, e.g. an EMCCD camera, a CMOS camera or any suitable imager.
  • a photodetector e.g. an EMCCD camera, a CMOS camera or any suitable imager.
  • the fluorescent radiation may be detected and the photon count per detector pixel may be stored in association with the excitation pattern.
  • the steps of illuminating the sample with an excitation pattern and detecting the fluorescent radiation may be repeated for at least one full series of excitation patterns. In an embodiment, measurements may be repeated over a plurality of series of excitation patterns.
  • excitation patterns may be re-optimised 516, based on the detected radiation.
  • Re-optimising may give a scaling proportional to x W2 where k is the number of iterations as described in Gwosch et al.
  • the activation and excitation patterns may be shifted 512 relative to the sample.
  • the sample may be shifted, in another embodiment, the patterns may be shifted by e.g. manipulation of the light travel path.
  • the stored photon counts and excitation pattern information may be used to reconstruct 510 a localization image of the sample, for example according to the method explained by J. Cnossen et al. (discussed in the background section and with reference to Fig. 1).
  • a priori information about the activation pattern may be used to localize fluorescent molecules in the sample and/or to reconstruct a localization image, for instance using maximum likelihood estimation and/or using the method detailed by J. Cnossen et al.
  • the background radiation may be detected 514 to optimize the excitation patterns, e.g. to optimize the phase shift between excitation patterns in a partial series of excitation patterns sharing a same orientation.
  • Fig. 6 schematically depicts a structured illumination microscopy system according to an embodiment of the invention.
  • the system may comprise a first optical module, an excitation module, for generating a sinusoidally varying excitation light pattern in the plane of the sample and a second optical module, an activation module, for generating an activation light pattern.
  • the first and second optical modules may be controlled by a computer to enhance the lateral localization precision of the system.
  • the excitation module may generate a sinusoidally varying excitation light pattern using a first light source 602 and computer-controlled spatial light modulator 612.
  • the spatial light modulator may be configured as a grating that is capable of producing excitation light patterns of a desired phase and orientation.
  • the first light source 602 preferably a single mode coherent source, e.g. a 640 nm laser, may be used to generate an excitation light beam, which may be expanded using a telescope 604,606 and guided via a polarizing beam splitter (PBS) 608 and a half wave plate (HWP) 610 towards the SLM.
  • PBS polarizing beam splitter
  • HWP half wave plate
  • the polarizing beam splitter and a half wave plate may be used to match the chief ray to the normal axis of the spatial light modulator.
  • Each pixel of the SLM behaves as a half-wave plate with a fast axis either vertical or at 45 degree depending on the state at which each pixel is programmed.
  • a plane wave reflected off the SLM is phase modulated by pi between the pixels in two different states.
  • Modulated light reflected from the SLM may be projected via the PBS to a mask 616 which may be used to transmit the +/- 1 diffraction order and block (higher) stray orders. This way, the modulation contrast may be improved.
  • a zero-vortex half wave plate (ZV-HWP) 618 may be placed after the mask. This way, the polarisation may be aligned with the direction of the interference pattern to increase contrast and reduce background light, resulting in a zero-point in the excitation light pattern where the intensity is as close to zero as possible.
  • the activation module may include a second light source 632, e.g. a 405 nm laser, for generating an activation light beam.
  • the activation light beam may be expanded using a suitable optical system 634,636 and guided towards a microlens array 638, which may be configured to generate an array of focused light spots (foci) 640.
  • the generation of the array of light spots is described in more detail in e.g. A.G.York et al., Instant Super-Resolution Imaging in Live Cells and Embryos via Analog Image Processing" , Nature Methods 10:11 (2013) pages 1122-1126.
  • the contrast of the focused light spots may be increased using an activation mask 642 comprising an array of holes that match the array of light spots.
  • Precise positioning of the generated activation light pattern relative to the excitation light pattern may be accomplished using a suitable optical system 644,646 in combination with an actuator-controlled mirror 648, e.g. a piezo-electric MEMS mirror.
  • an actuator-controlled mirror 648 e.g. a piezo-electric MEMS mirror.
  • the use of the microlens array in combination with an actuator-controlled mirror allows accurate positioning ( ⁇ 1 nm) and high contrast (>80%). This way, it is possible to position the activation areas relative to the position of the minima of the sinusoidal light patterns with great accuracy.
  • the light paths of the excitation light beam 619 and the activation light beam 645 may be combined using a first dichroic mirror 620 and may be further guided to a sample holder 624 using a second dichroic mirror 622. Fluorescent light emitted by a sample on the sample holder may travel through the second dichroic mirror to a camera 626, which may be used to record a fluorescent image of the sample.
  • a computer 600 connected to the system may execute a software program executing program codes for controlling the different elements of the SIM system.
  • the computer may control the first light source and the SLM of the excitation light module, the second light source and the actuator-controlled mirror of the activation light module and the camera for recording the sample during the exposure of a sample to activation patterns and excitation patterns.
  • the computer may execute a program for executing the steps as described with reference to Fig. 5.
  • the first and second light sources may be digitally modulated (on and off) at a frequency between 100 and 400 MHz, for example 250 MHz.
  • Fig. 6 only depicts an exemplary non-limiting embodiment of a pattern-activated SIM system and many other implementations and variants are possible without departing from the inventive concept.
  • an SLM configured as a grating
  • Fig. 7A and 7B schematically illustrate the gain in localization accuracy that may be achieved using the embodiments described in this application.
  • Fig. 7A shows a gain in localization precision of the known SIMFLUX method as discussed in more detail with reference to Fig.lA and 1B, compared to conventional Single Molecule Localisation Microscopy (SMLM).
  • SIMFLUX Single Molecule Localisation Microscopy
  • a series of phase-shifted excitation light patterns 702 I-3 may be used to illuminate a part of the sample and excite fluorescent molecules in that part.
  • Equidistance phase shifts may be selected such that the total or cumulative excitation light 704 is substantially constant over the exposed sample region. This may lead to an improvement factor (IF) that varies between roughly 2.0 and 2.4, with an average of 2.20. If photoactivatable fluorescent molecules would be used, a homogeneous constant activation pattern would be applied, and this would not affect the improvement factor.
  • IF improvement factor
  • Fig. 7B schematically shows a gain in localization precision for structured illumination microscopy methods according to the embodiments described in this application.
  • the improvement is compared to conventional Single Molecule Localisation Microscopy (SMLM).
  • SMLM Single Molecule Localisation Microscopy
  • the final improvement may depend on e.g. the shape of the activation pattern, the shape of the excitation patterns, the distance between excitation patterns, and the background noise.
  • the improvement factor varies between roughly 2.2 and 7.8, with an average improvement factor of 3.14.
  • Fig. 8 schematically depicts the pattern activated structured illumination microscopy method according to an embodiment of the invention.
  • a sample comprising a photo-activatable fluorescent molecule 802 is illuminated with activation light in a first illumination pattern 806. Only one activation area is visible in the depicted frame 804.
  • the activation pattern has high intensity areas (light) and low intensity areas (dark).
  • an image 810 may be recorded, but this image does not comprise localisation information.
  • the sample is illuminated with excitation light in a series of three sinusoidal wave intensity patterns 808 I-3 with 3 phase shifted patterns with equal phase steps of 45° per orthogonal orientation of the line pattern (only one orientation is shown).
  • the excitation pattern has high intensity areas (light) and low intensity areas (dark).
  • the fluorescent radiation is detected and an image is recorded, resulting in three images 812 I-3 for a first orientation of the line pattern.
  • Each image comprises pixels, where the pixel value represents the photon count. A higher photon count is represented with a lighter colour. Even in the absence of a fluorescent molecule in a focal plane, some radiation may be detected, resulting in background noise.
  • First localization information may be obtained by estimating the centroid of the bright spots, i.e. the regions with a photon count higher than the background noise.
  • the estimated centroid location is indicated by a circle 814 I-3 , with the size of the circle proportional to the uncertainty in centroid location.
  • Second localization information may be obtained by correlating the photon count with the excitation pattern.
  • a high photon count is associated with a high excitation light intensity, and a low photon count with a low excitation light intensity.
  • the photon counts for the three images per orientation are known to be spaced (in this case) 45° apart. As the position of the excitation pattern is known, this provides additional a priori information (modulo the wavelength of the excitation pattern).
  • Third localisation information may be obtained by correlating the photon count with the activation pattern.
  • a high cumulative photon count is associated with a high activation light intensity, and a low photon count with a low activation light intensity.
  • this provides additional a priori information (modulo the period of the activation pattern).
  • the first localization information and the second localization information and the third localization information may be combined to effectively increase the localization precision.
  • the same procedure may be executed with a second series of excitation patterns in a different direction, e.g. orthogonal to the first series of excitation patterns, to increase the localization precision in several directions.
  • the measurement may be repeated several times.
  • the measurements may then be combined into a localization image 816, in which the estimated location of the fluorescent molecule for each measurement is shown with a dot 820 and a circle 818 represents the combined uncertainty.
  • the localisation uncertainty as may be obtained with the SIMFLUX method discussed with reference to Fig. 1 is displayed with a dashed circle 822.
  • the Pattern Activated SIMFLUX method may result in an improvement of localization precision with a factor of about 1.4-3.4, compared to the known SIMFLUX method, and with a factor of about 3.14-7.5 compared to conventional position estimation with uniform illumination, i.e. Single Molecule Localisation Microscopy (SMLM).
  • SMLM Single Molecule Localisation Microscopy
  • Fig. 9 schematically depicts part of a structured illumination microscopy system according to an embodiment of the invention.
  • the figure depicts a further (third) optical module that may be used for axial localization precision enhancement.
  • This module may be used separately or may be (coherently) combined with the optical modules described with reference to Fig. 6, which are configured for lateral localization precision enhancement.
  • This module generates a third illumination pattern, which may be an orthogonal pattern relative to the sample plane (with maximal pitch in the z direction) using two counter-propagating plane waves.
  • a 3D pattern-activated (PA) SIM system can be realized, which is adapted to generate 3D orthogonal illumination and activation patterns in a sample and to use these patterns to achieve localization precision improvement in the lateral and axial directions according to the general teaching of this application.
  • PA pattern-activated
  • the optical module may include a light source 902, preferably a single mode coherent source, e.g. a 640 nm laser, which may be used to generate an excitation light beam, which may be directed to a first polarizing beam splitter (PBS) 904i, which splits the excitation light beam in two coherent beams, a first excitation light beam 906i and a second excitation light beam 906 2.
  • a light source 902 preferably a single mode coherent source, e.g. a 640 nm laser, which may be used to generate an excitation light beam, which may be directed to a first polarizing beam splitter (PBS) 904i, which splits the excitation light beam in two coherent beams, a first excitation light beam 906i and a second excitation light beam 906 2.
  • PBS polarizing beam splitter
  • the first excitation light beam may be guided towards a second polarizing beam splitter 904 2 which relays the first beam towards a first optical lens system 907i for expanding the beam to predetermined cross-sectional dimensions and further guiding the beam via a third polarizing beam splitter 904 3 toward the sample 914.
  • the second excitation light beam may be guided towards a fourth polarizing beam splitter 904 4 which relays the second beam 906 2 towards a second optical lens system 907 2 for expanding the beam to predetermined cross-sectional dimensions and exposing the sample with plane waves from a second direction, which is the counter-direction of the first beam.
  • the beams may interfere at the position of the sample 914 forming a sinusoidal light pattern in the axial direction of the sample.
  • Two actuator-controlled mirrors 910 I , 2 may be used to shift the pattern in the axial direction with high accuracy ( ⁇ 0.1 nm). Alternatively, electric optical modulators may be used. Further, first and second actuator-controlled mirrors 912 1 2 , e.g. MEMS mirrors, may be configured to accurately position the first light beam relative to the second light beam.
  • the first light beam may be directed to the sample 914 using a dichroic mirror 916; the second light beam may be directed on the sample from a direction opposite from the first light beam.
  • the first light beam and the second light beam create an interference pattern 922 in the axial extent of the sample, the dashed lines indicating the depth of focus.
  • small volumes e.g. 850 nm
  • an activation laser (405 nm) over the depth of focus.
  • the same activation pattern may be used as in 2D pattern activated structured illumination localization microscopy, while in other embodiments, a different activation pattern may be used.
  • a so-called 3D “woodpile” pattern may be used.
  • Such a 3D woodpile patterns may e.g. be formed interfering two oblique plane waves with a counter-propagating axial plane wave.
  • the excitation light source and the actuator-controlled mirrors may be connected to a computer which is configured to execute a 3D pattern-activated SIM scheme.
  • the activation patterns and/or excitation patterns may be three- dimensional patterns.
  • Three dimensional activation and excitation patterns may be implemented using a single objective or multiple objectives.
  • 3D activation and excitation patterns may also be generated using light sheets, for example as shown in B.-J. Chang et al., ‘Universal light-sheet generation with field synthesis’, Nature Methods, volume 16 (2019), pages 235-238.
  • the upper beam path of the set-up shown in Fig. 9, comprising second beam 906 2 and the optics along the second beam path, up to and including actuator-controlled mirror 912i may be replaced with a piezo-mounted mirror, reflecting the first light beam after it has traversed the sample.
  • a low numerical aperture objective and a tube lens may be added to the beam path between the sample and the mirror to improve the optics.
  • a similar set-up is known from J.D. Manton et al., ‘Structured illumination microscopy with extended axial resolution through mirrored illumination’, arXiv:1810.04590v1 , Fig. 2d (part below the sample).
  • a difference with the set-up as shown by J.D. Manton et al. is that the mirror is mounted on a piezo stage to shift the pattern in the axial direction.
  • Fig. 10A-10C depict the localization improvement in the axial direction for a pattern-activation structured illumination microscopy scheme compared to a conventional microscopy scheme, in particular a conventional single-molecule localization microscopy scheme.
  • the figures show a 12-time improvement when compared to conventional astigmatism.
  • the counter propagating waves may be shifted with 3 different phase steps, to create three phase-shifted sinusoidal excitation light patterns.
  • Fig. 10B depicts a conventional 3D localization method is using an astigmatic lens which is configured to generate a point spread function (PSF) that changes shape in the axial direction.
  • PSF point spread function
  • FIG. 10B depicts the SIMFLUX axial theoretical localization precision 1002i (calculated using the Cramer Rao Lower Bound (CRLB)) and the CRLB of a 3D localization 1002 2 using an astigmatic lens.
  • Fig. 10C depicts the improvement factor 1006 (the ratio of CRLBs) together with the activation intensity 1004 as function of axial position. Interpreting the activation intensity as the probability of activating a molecule, we can multiply the improvement factor and the activation intensity. This shows an average localization precision improvement of 12 over conventional 3D localization using an astigmatic lens.
  • Fig. 11 is a block diagram illustrating an exemplary data processing system that may be used in embodiments as described in this disclosure.
  • Data processing system 1100 may include at least one processor 1102 coupled to memory elements 1104 through a system bus 1106. As such, the data processing system may store program code within memory elements 1104. Furthermore, processor 1102 may execute the program code accessed from memory elements 1104 via system bus 1106.
  • data processing system may be implemented as a computer that is suitable for storing and/or executing program code. It should be appreciated, however, that data processing system 1100 may be implemented in the form of any system including a processor and memory that is capable of performing the functions described within this specification.
  • Memory elements 1104 may include one or more physical memory devices such as, for example, local memory 1108 and one or more bulk storage devices 1110.
  • Local memory may refer to random access memory or other non- persistent memory device(s) generally used during actual execution of the program code.
  • a bulk storage device may be implemented as a hard drive or other persistent data storage device.
  • the processing system 1100 may also include one or more cache memories (not shown) that provide temporary storage of at least some program code in order to reduce the number of times program code must be retrieved from bulk storage device 1110 during execution.
  • I/O devices depicted as input device 1112 and output device 1114 optionally can be coupled to the data processing system.
  • input device may include, but are not limited to, for example, a keyboard, a pointing device such as a mouse, or the like.
  • output device may include, but are not limited to, for example, a monitor or display, speakers, or the like.
  • Input device and/or output device may be coupled to data processing system either directly or through intervening I/O controllers.
  • a network adapter 1116 may also be coupled to data processing system to enable it to become coupled to other systems, computer systems, remote network devices, and/or remote storage devices through intervening private or public networks.
  • the network adapter may comprise a data receiver for receiving data that is transmitted by said systems, devices and/or networks to said data and a data transmitter for transmitting data to said systems, devices and/or networks.
  • Modems, cable modems, and Ethernet cards are examples of different types of network adapter that may be used with data processing system 1100.
  • memory elements 1104 may store an application 1118. It should be appreciated that data processing system 1100 may further execute an operating system (not shown) that can facilitate execution of the application. Application, being implemented in the form of executable program code, can be executed by data processing system 1100, e.g., by processor 1102. Responsive to executing application, data processing system may be configured to perform one or more operations to be described herein in further detail.
  • data processing system 1100 may represent a client data processing system.
  • application 1118 may represent a client application that, when executed, configures data processing system 1100 to perform the various functions described herein with reference to a "client".
  • client can include, but are not limited to, a personal computer, a portable computer, a mobile phone, or the like.
  • data processing system may represent a server.
  • data processing system may represent a server, a cloud server or a system of (cloud) servers.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Chemical & Material Sciences (AREA)
  • Analytical Chemistry (AREA)
  • Optics & Photonics (AREA)
  • Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biochemistry (AREA)
  • General Health & Medical Sciences (AREA)
  • Immunology (AREA)
  • Pathology (AREA)
  • Microscoopes, Condenser (AREA)
  • Investigating, Analyzing Materials By Fluorescence Or Luminescence (AREA)

Abstract

Module optique pour microscopie à lumière structurée et procédé de microscopie à lumière structurée, le procédé comprenant l'éclairage d'une zone d'exposition d'un échantillon comprenant des molécules fluorescentes photoactivables avec un motif lumineux d'activation, le motif d'activation comprenant un ou plusieurs maxima de lumière d'activation, la lumière d'activation activant une ou plusieurs des molécules fluorescentes photoactivables ; l'éclairage de manière séquentielle de la zone d'exposition avec une pluralité de motifs de lumière d'excitation, la pluralité de motifs de lumière d'excitation définissant un motif de lumière d'excitation cumulative, le motif de lumière d'excitation cumulative définissant un ou plusieurs minima de lumière d'excitation cumulative, le motif d'excitation cumulé étant positionné par rapport au motif d'activation de telle sorte que chaque maximum du motif de lumière d'activation coïncide avec l'un du ou des minima de lumière d'excitation cumulatifs, la lumière d'excitation d'un motif de lumière d'excitation excitant une ou plusieurs des molécules fluorescentes activées ; la détection d'un rayonnement émis par les molécules fluorescentes photoactivables qui sont activées par le motif de lumière d'activation et exciter la pluralité de motifs de lumière d'excitation ; et la détermination des informations de position d'une ou de plusieurs des molécules fluorescentes photoactivables sur la base du rayonnement détecté.
PCT/EP2020/076337 2019-09-19 2020-09-21 Microscopie de localisation d'éclairage structurée activée par un motif WO2021053245A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
NL2023860A NL2023860B1 (en) 2019-09-19 2019-09-19 Pattern activated structured illumination localization microscopy
NL2023860 2019-09-19

Publications (1)

Publication Number Publication Date
WO2021053245A1 true WO2021053245A1 (fr) 2021-03-25

Family

ID=68425229

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2020/076337 WO2021053245A1 (fr) 2019-09-19 2020-09-21 Microscopie de localisation d'éclairage structurée activée par un motif

Country Status (2)

Country Link
NL (1) NL2023860B1 (fr)
WO (1) WO2021053245A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP4075180A1 (fr) * 2021-04-13 2022-10-19 Abberior Instruments GmbH Procédé, dispositif et programme informatique pour déterminer une position d'au moins un émetteur dans un échantillon
CN115825032A (zh) * 2023-02-08 2023-03-21 之江实验室 一种数字化荧光仿生模体成像方法及系统

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090237501A1 (en) * 2008-03-19 2009-09-24 Ruprecht-Karis-Universitat Heidelberg Kirchhoff-Institut Fur Physik method and an apparatus for localization of single dye molecules in the fluorescent microscopy
WO2014046606A1 (fr) * 2012-09-24 2014-03-27 Ge Healthcare Bio-Sciences Corp. Procédés de résolution de positions en microscopie stochastique de fluorescence au moyen d'un éclairage à structure tridimensionnelle
WO2017153430A1 (fr) 2016-03-07 2017-09-14 MAX-PLANCK-Gesellschaft zur Förderung der Wissenschaften e.V. Procédé d'imagerie locale à haute résolution d'une structure dans un échantillon, afin de détecter des réactions d'un objet présentant un intérêt à des conditions environnementales modifiées
US10247672B2 (en) 2014-09-29 2019-04-02 Howard Hughes Medical Institute Non-linear structured illumination microscopy

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090237501A1 (en) * 2008-03-19 2009-09-24 Ruprecht-Karis-Universitat Heidelberg Kirchhoff-Institut Fur Physik method and an apparatus for localization of single dye molecules in the fluorescent microscopy
WO2014046606A1 (fr) * 2012-09-24 2014-03-27 Ge Healthcare Bio-Sciences Corp. Procédés de résolution de positions en microscopie stochastique de fluorescence au moyen d'un éclairage à structure tridimensionnelle
US10247672B2 (en) 2014-09-29 2019-04-02 Howard Hughes Medical Institute Non-linear structured illumination microscopy
WO2017153430A1 (fr) 2016-03-07 2017-09-14 MAX-PLANCK-Gesellschaft zur Förderung der Wissenschaften e.V. Procédé d'imagerie locale à haute résolution d'une structure dans un échantillon, afin de détecter des réactions d'un objet présentant un intérêt à des conditions environnementales modifiées

Non-Patent Citations (9)

* Cited by examiner, † Cited by third party
Title
A.G.YORK ET AL.: "Instant Super-Resolution Imaging in Live Cells and Embryos via Analog Image Processing", NATURE METHODS, vol. 10, no. 11, 2013, pages 1122 - 1126, XP055554918, DOI: 10.1038/nmeth.2687
B.-J. CHANG ET AL.: "Universal light-sheet generation with field synthesis", NATURE METHODS, vol. 16, 2019, pages 235 - 238, XP036725909, DOI: 10.1038/s41592-019-0327-9
BALZAROTTI ET AL.: "Nanometer resolution imaging and tracking of fluorescent molecules with minimal photon fluxes", SCIENCE, vol. 355, pages 606 - 612, XP055474328, DOI: 10.1126/science.aak9913
CNOSSEN ET AL., LOCALIZATION MICROSCOPY AT DOUBLE PRECISION WITH PATTERNED ILLUMINATION
GWOSCH ET AL., MINFLUX NANOSCOPY DELIVERS MULTICOLOR NANOMETER 3D-RESOLUTION IN (LIVING) CELLS, Retrieved from the Internet <URL:https://www.biorxiv.rg/content/10.1101/734251v1>
J. CNOSSEN ET AL.: "Localization microscopy at double precision with patterned illumination", BIORXIV 554337
J.D. MANTON ET AL.: "Structured illumination microscopy with extended axial resolution through mirrored illumination", ARXIV:1810.04590V1
PALM, BETZIG ET AL.: "Imaging Intracellular Fluorescent Proteins at Nanometer Resolution", SCIENCE, vol. 313, 2006, pages 1643 - 1645
STORM, RUST ET AL.: "Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM", NATURE METHODS, vol. 3, 2006, pages 793 - 795

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP4075180A1 (fr) * 2021-04-13 2022-10-19 Abberior Instruments GmbH Procédé, dispositif et programme informatique pour déterminer une position d'au moins un émetteur dans un échantillon
WO2022218834A1 (fr) * 2021-04-13 2022-10-20 Abberior Instruments Gmbh Procédé, dispositif et programme informatique pour déterminer une position d'au moins un émetteur dans un échantillon
CN115825032A (zh) * 2023-02-08 2023-03-21 之江实验室 一种数字化荧光仿生模体成像方法及系统
CN115825032B (zh) * 2023-02-08 2023-05-02 之江实验室 一种数字化荧光仿生模体成像方法及系统

Also Published As

Publication number Publication date
NL2023860B1 (en) 2021-05-25

Similar Documents

Publication Publication Date Title
Cnossen et al. Localization microscopy at doubled precision with patterned illumination
US9874737B2 (en) Method and apparatus for combination of localization microscopy and structured illumination microscopy
Gregor et al. Image scanning microscopy
US10795144B2 (en) Microscopy with structured plane illumination and point accumulation for imaging and nanoscale topography
JP6059190B2 (ja) 分解能の向上した顕微鏡法および顕微鏡
RU2660301C2 (ru) Флуоресцентная микроскопия высокого разрешения со структурированным пучком возбуждения
US10352860B2 (en) Super resolution microscopy
Ward et al. Image scanning microscopy: an overview
Squire et al. Three dimensional image restoration in fluorescence lifetime imaging microscopy
JP6393474B2 (ja) 高分解能３ｄ位置特定顕微鏡検査方法
JP2016534389A (ja) 開口走査フーリエタイコグラフィ撮像
JP5120873B2 (ja) 分光計測装置及び分光計測方法
JP2016530567A (ja) 可変照明フーリエタイコグラフィー撮像装置、システム、及び方法
JP2019534998A (ja) ルミネセンス光を放射するように励起光により励起可能な、試料中の単離分子の位置を空間的に高分解能に確定するための方法
US20180032896A1 (en) Method and system for quantum information processing and computation
US9921161B1 (en) Structured light active localization microscopy
JP7181537B2 (ja) 高空間分解能の時間分解イメージング法
JP2012515930A (ja) 広視野の超解像顕微鏡を提供するためのシステム、方法及びコンピューターがアクセス可能な媒体
Sandmeyer et al. DMD-based super-resolution structured illumination microscopy visualizes live cell dynamics at high speed and low cost
WO2021053245A1 (fr) Microscopie de localisation d&#39;éclairage structurée activée par un motif
Zeng et al. Computational methods in super-resolution microscopy
JP7472134B2 (ja) 試料の３次元領域の蛍光信号を検出するための方法および装置
Pawlowska et al. Embracing the uncertainty: the evolution of SOFI into a diverse family of fluctuation-based super-resolution microscopy methods
US10585272B2 (en) Coherent fluorescence super-resolution microscopy
RU2579640C1 (ru) Конфокальный спектроанализатор изображений

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20780128

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20780128

Country of ref document: EP

Kind code of ref document: A1