Classifications

• • G—PHYSICS
  • G02—OPTICS
  • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
  • G02B21/00—Microscopes
  • G02B21/0004—Microscopes specially adapted for specific applications
  • G02B21/002—Scanning microscopes
  • G02B21/0024—Confocal scanning microscopes (CSOMs) or confocal "macroscopes"; Accessories which are not restricted to use with CSOMs, e.g. sample holders
  • G02B21/008—Details of detection or image processing, including general computer control
• • G—PHYSICS
  • G02—OPTICS
  • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
  • G02B21/00—Microscopes
  • G02B21/0004—Microscopes specially adapted for specific applications
  • G02B21/002—Scanning microscopes
  • G02B21/0024—Confocal scanning microscopes (CSOMs) or confocal "macroscopes"; Accessories which are not restricted to use with CSOMs, e.g. sample holders
  • G02B21/0052—Optical details of the image generation
  • G02B21/0076—Optical details of the image generation arrangements using fluorescence or luminescence
• • G—PHYSICS
  • G02—OPTICS
  • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
  • G02B21/00—Microscopes
  • G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
  • G02B21/365—Control or image processing arrangements for digital or video microscopes
  • G02B21/367—Control or image processing arrangements for digital or video microscopes providing an output produced by processing a plurality of individual source images, e.g. image tiling, montage, composite images, depth sectioning, image comparison

Definitions

• the invention relates to structured illumination microscopy, and, in particular, though not exclusively, to methods and systems for structured illumination scanning microscopy and a computer program product for executing such methods .
• Super resolution microscopy methods are known for resolving spatial resolutions beyond the diffraction limit.
• An example of such a super-resolution method is the well-known structured illumination microscopy (SIM) as e.g. described in the article by Gustafsson et al, Journal of Microscopy, Vol. 198, Pt 2, May 2000, pp. 82-87.
• SIM structured illumination microscopy
• a series of light patterns also referred to as structured light
• bright maxima associated with high light intensities
• dark minima associated with low light intensities
• Recorded images are subsequently processed to extract the high-resolution information and construct a super-resolution image, i.e. an image having a resolution surpassing the diffraction limit.
• SIM is to obtain a super-resolution image of an unknown sample structure - or more precisely - of an unknown spatial distribution of fluorescent dyes.
• Structured illumination light typically generated using a diffraction grating, is incident on the spatial distribution of
• the fluorescent dyes causes the fluorescent dyes to emit emission light.
• the intensity of emission light originating from a point on the sample is proportional to the product of dye concentration at this point and the effective light intensity of the structured light at this point.
• the resulting emission light is thus a product of two patterns, namely the structured light and the light of an (unknown) distribution of fluorescent molecules. As a result, the emission light will contain observable Moire effects from which normally
• the light pattern typically includes a two or three dimensional arrangement of parallel lines in one or more planes perpendicular to the optical axes. This light pattern is used to expose a sample several times, wherein after each exposure, the light pattern is shifted and/or rotated. Recoded images of the emission light of each exposure are used to reconstruct a high resolution image using a known mathematical image reconstruction algorithm.
• An example of such method is described by Lai et. al , in document arXiv : 1602.06904vl
• a 2D arrangement of parallel lines may be represented by a sinusoid having an amplitude, a modulation depth, an initial phase, a direction and a spatial frequency, wherein the amplitude relates to the light intensity of the bright maxima and dark minima of a light pattern and the modulation depth relates to the difference in light
• the spatial frequency defines the number of bright maxima per distance perpendicular to the line- direction.
• the initial phase defines the position of a light patterns on a sample relative to each other. For reconstructing a high-resolution image it is important to know the parameters of the structured light pattern as accurate as possible .
• parameters such as the phase information and the modulation depth
• image analysis algorithms can only retrieve these parameters if the recorded images comprise a clear enough "footprint" of the used structured light pattern. This means that the recorded images must comprise discernible bright maxima and dark minima corresponding with the maxima and minima of the used structured light pattern otherwise the algorithms cannot retrieve the positions of the bright maxima on the sample.
• Emission light passes through at least one NA- limiting optical element (e.g. an objective lens) before being incident on an image sensor.
• NA-limiting optical element e.g. an objective lens
• such NA-limiting element acts as a spatial low pass filter in the sense that it attenuates the amplitudes of high spatial frequency components.
• the spatial frequency of an applied structured light pattern is too high, i.e. if the structured light pattern is too fine, the amplitude associated with this frequency may be attenuated significantly by the lens system.
• the bright maxima and dark minima corresponding to the structured light pattern may only be weakly present in a recorded image.
• the footprint of the structured light pattern may then not be discernible anymore by the image processing algorithms, which may impede
• the lens system thus imposes an upper limit to the spatial frequency that can be used for structured light patterns. This is disadvantageous, since the spatial frequency of the applied structured light patterns are preferably as high as possible, because finer structured light patterns yield higher resolutions.
• US 2013/0314717 Al discloses a SIM system that includes a modulator to modulate the light such that the sample is exposed to structured light. In this scheme, a modulated focal spot of light is scanned over the sample.
• the modulation is controlled such that the sample is exposed to a 2D pattern of parallel lines wherein the lines make an angle with the scanning direction.
• the scanning movement of the focal spot over the sample must be accurately timed with respect the modulation in order to generate an accurate 2D pattern of parallel lines of a high spatial frequency.
• the parameters of the light pattern are derived from the captured images.
• aspects of the present invention may be embodied as a system, method or computer program product. Accordingly, aspects of the present invention may take the form of an entirely
• aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium (s) having computer readable program code embodied, e.g., stored, thereon.
• the computer readable medium may be a computer readable signal medium or a computer readable storage medium.
• a computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.
• a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
• a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro ⁇ magnetic, optical, or any suitable combination thereof.
• a computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
• Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber, cable, RF, etc., or any suitable combination of the foregoing.
• Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java(TM), Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming
• the program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server.
• the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN) , or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider) .
• Computer program instructions may be provided to a processor, in particular a microprocessor or central processing unit (CPU) , of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer, other programmable data processing apparatus, or other devices create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks .
• a processor in particular a microprocessor or central processing unit (CPU)
• CPU central processing unit
• These computer program instructions may also be stored in a computer readable medium that can direct a
• the computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
• each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function (s) .
• the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.
• SSIM scanning structured illumination microscopy
• the invention relates to a method of forming a high-resolution image of a sample using a scanning microscope controlled by a processor.
• the method may include: the processor receiving or generating control information for controlling a scanning microscope, the control information defining a plurality of different periodic patterns on the basis of one or more pattern parameters; the processor using the control information to control the scanning microscope to expose the sample to multiple illumination patterns, each exposure to an illumination pattern causing one or more optical excitations in the sample, the light originating from said optical excitations forming an emission light signal; the processor controlling an imaging system to capture multiple images, each image being associated with an emission light signal of one of the multiple exposures, and, the processor using an image reconstruction algorithm for forming a high- resolution image on the basis of at least one of the one or more pattern parameters and the captured images.
• the one or more pattern parameters may include at least one of: a spatial frequency, a periodicity direction, an initial phase, an illumination intensity.
• the method according to the invention uses control information defining a plurality of different periodic patterns on the basis of one or more pattern parameters which is used by a processor, e.g. a computer, of the scanning microscope to expose a sample to multiple different period illumination patterns, i.e. a light pattern that provides a certain light distribution, e.g. a 2D light distribution or a 3D light distribution, within the sample.
• a processor e.g. a computer
• the scanning microscope can accurately control the position of a focused illumination light spot on the sample, a periodic 2D or 3D illumination pattern can be accurately written into the sample using the control information.
• the focused illumination light spot may comprise a light intensity profile, for example a Gaussian intensity profile, so that areas of the pattern may be understood to have been illuminated with a light intensity exceeding a predetermined threshold light intensity.
• the intensity of light may relate to the physical quantity irradiance.
• the parameters defining the periodic patterns - which are determined before exposing the sample - may be used in the reconstruction algorithm.
• the invention thus eliminates or at least substantially reduces the need for a posteriori image analysis in order to determine illumination light pattern parameters that are used by the reconstruction algorithm. This way, light patterns may be used that have a higher spatial frequency than the light patterns that are used in SIM systems known in the prior art.
• controlling the scanning microscope may include controlling a scanning mirror of the scanning microscope to move at least one focused illumination light spot through the sample positioned a sample holder.
• the position of the sample and the light spot may be determined on the basis of a reference position associated with the sample or the sample holder.
• a reference position associated with the sample (e.g. on the sample) or the sample holder may be used by the scanning microscope in order to determine the position of the focused illumination light spot in the sample.
• exposing the sample to multiple illumination patterns may include moving the position of a focused illumination light spot of the scanning microscope in accordance with each of said multiple periodic patterns.
• a reference position associated with the sample may be used to define a coordinate system, which may be used by the processor to move the illumination light spot to predetermined positions relative to the
• the relative position of the different illumination patterns with respect to the reference position are determined on the basis of a calibration of an optical system, wherein the calibration links settings of the optical system, such as respective positions of said scanning mirror, to positions of the illumination light spot on the sample.
• a periodic pattern associated with certain pattern parameters may be defined as a set of coordinates on the sample, which the processor may translate into a set of mirror positions for moving the focused illumination light spot in accordance with the pattern.
• the method comprises the processor obtaining calibration data.
• the calibration data associate a first state of the optical system to a region of an image of a sample, e.g. a reference sample, captured by the imaging system.
• the region represents a part of the sample that is exposed to illumination when the optical system is in said first state.
• the region may comprise or consist of one or more image pixels.
• the reference sample may be the same sample of which a high-resolution image is formed in accordance with methods described herein.
• states of the optical system may relate to respective positions and/or orientations of one or more movable elements of the optical system.
• Examples of a movable element of an optical system are a scanning mirror, a rescanning mirror, and a sample holder, such as a sample stage. These moveable elements may be movable with respect to a surface supporting the microscope and their movement may be controllable.
• the region is discernable in the captured image.
• obtaining the calibration data comprises identifying said region in the captured image.
• Identifying the region may comprise identifying aberrant image pixel values, for example relatively bright image pixel values in the image, the aberrant image pixel values constituting said image region.
• identifying said region in the captured image may comprise executing a phase retrieval algorithm known in the art.
• the scanning microscope may be a scanning microscope in the sense that it may be configured to perform sample scanning.
• the sample may be positioned in a sample holder that can be controlled to move with respect to a focused illumination light spot, wherein the focused illumination light spot is fixed relative to a surface supporting the scanning
• emission light may be collected by an imaging system such as an array detector or camera, and an image of the sample may be constructed based on the emission light that is captured while scanning the sample point by point by moving the sample with respect to the fixed illumination light spot.
• exposing a sample to an illumination pattern thus comprises repeatedly
• a state of the optical system described above may relate to a position and/or orientation of the sample holder relative to the focused illumination light spot.
• the processor uses the image reconstruction algorithm for forming the high-resolution image of the sample on the basis of the calibration data.
• the calibration data may namely be valid for any image captured by the optical system irrespective of what kind of sample is imaged.
• the calibration data may associate the exact center of a calibration image with a first state of the system. Then, if the optical system captures a particular image of an actual sample and in this process adopts the first state, then, in accordance with the calibration data, the first state of the optical system is associated with the exact center of the particular image, which means that the exact center of the particular image represents a part of the actual sample, which part was exposed to illumination when the optical system was in the first state.
• the calibration data may be understood to define a reference position
• the processor obtaining calibration data comprises the processor controlling the optical system to adopt a first state herewith controlling the scanning microscope to expose a first part of the reference sample to a focused illumination light spot.
• This embodiment further comprises the processor controlling the imaging system to capture a calibration image of the reference sample, the calibration image comprising a first region representing said exposed first part of the reference sample.
• This embodiment further comprises the processer storing the first state of the optical system in association with the first region of the calibration image.
• the calibration data associate a plurality of states of the optical system, e.g. a plurality of positions and/or orientations of a scanning mirror of the optical system, to respective regions of one or more images of the reference sample captured by the imaging system, wherein each particular region of said respective regions represents a part of the reference sample that is exposed to illumination when the optical system is in the state associated with the particular region.
• obtaining the calibration data comprises the processor controlling the optical system to adopt a second state, e.g. to control the at least one
• This embodiment comprises the processor controlling the imaging system to capture one or more calibration images, at least one calibration image of the one or more calibration images comprising the first region representing said exposed first part of the reference sample and at least one calibration image of the one or more
• calibration images comprising a second region representing the second exposed part of the reference sample.
• This embodiment also comprises the processer storing the second state of the optical system in association with the second region.
• the optical system may be controlled to adopt the first and second state in the sense that the optical system is controlled to move the focused illumination light spot relative to the sample to expose the sample to a calibration illumination pattern.
• a single calibration image comprises both the first and the second region described above .
• controlling the optical system to adopt the first state, and optionally to adopt the second state comprises controlling an orientation and/or position of at least one movable element of the optical system, such as a scanning mirror and/or rescanning mirror and/or sample holder of the optical system.
• a movable element, such as a mirror or sample holder, in such an optical system may have a position and/or orientation that depends on a voltage that is applied to it, in particular that is applied to a control mechanism of the movable element, which control mechanism may comprise one or more actuators and/or electro motors.
• controlling an orientation and/or position of a movable element may comprise controlling a voltage that is applied to the movable element, in particular to a control mechanism of the movable element.
• the illumination light spot may comprise a predetermined light intensity pattern comprising at least two spatially arranged light intensity maxima.
• the spatially arranged light intensity maxima may define a spatial frequency of the illumination pattern.
• the at least two light intensity maxima may be arranged next to each other in a plane perpendicular to the optical axis of the microscope. In another embodiment, the at least two light intensity maxima may be arranged in an axial direction parallel to optical axis of the microscope.
• the predetermined light intensity pattern may comprise at least a 2D arrangement of two or more light intensity maxima; and/or, wherein the predetermined light intensity pattern comprises a 3D arrangement of multiple light intensity maxima.
• the illumination light spot may be a shaped illumination light spot, the shaped light spot comprising at least a first width and at least a second width small than the first width.
• the second width may be arranged perpendicular to the
• capturing multiple images may include controlling a rescanning mirror of the scanning microscope to reflect the emission light signal towards an imaging system and to move a focused emission light signal in accordance with the illumination pattern over the imaging plane of the imaging system.
• the optical response may comprise emission light that is emitted by fluorophores in the sample.
• the illumination light spot may excite the fluorophores and, as a result, the fluorophores may emit the emission light.
• the optical response may comprise a collimated emission light beam that is formed using an optical system, such as a lens system.
• controlling the scanning and the rescanning mirror may include rotating the scanning mirror and the rescanning mirror back and forth over a predetermined first angular amplitude and a second angular amplitude
• the ratio between the first and second angular amplitude may be unequal to one.
• the second angular amplitude may be selected two times the first angular amplitude.
• forming a high-resolution image further includes: determining a Fourier transform of the different multiple illumination patterns on the basis of the one or more pattern parameters .
• the scanning microscope may comprise at least a pinhole for filtering out-of-focus light out of the illumination light signal.
• the scanning microscope may comprise one or more optical elements configured to project an illumination light spot in a plane of interest of the sample onto a confocal conjugate plane of the plane of interest.
• the use of a pinhole may significantly increase the signal-to-noise ratio of the recorded image data. This is because the sample is exposed to a illumination pattern by scanning an illumination light spot over the sample. This way, a scanning structure illumination microscope may be realized that has confocal characteristics.
• control information may be adapted to control one or more actuators for controlling a scanning mirror and/or rescanning mirror in order to move a focused illumination light spot in a lateral direction in the sample; and/or, to control one or more optical elements, preferably one or more lenses, in order to move the focused illumination light spot in an axial direction in the sample.
• the invention may relate to a scanning microscopy system for forming a high-resolution image of a sample comprising: a computer adapted to control the scanning microscope, the computer comprising a computer readable storage medium having at least part of a program embodied therewith; and, a computer readable storage medium having computer readable program code embodied therewith, and a processor, preferably a microprocessor, coupled to the
• control information for controlling a scanning microscope, the control information defining a plurality of different periodic patterns on the basis of one or more pattern parameters, preferably the one or more pattern parameters, including at least one of: a spatial frequency, a periodicity direction, an initial phase; using the control information to control the scanning microscope to expose the sample to multiple illumination patterns, each exposure to an illumination pattern causing one or more optical excitations in the sample, the light originating from said optical
• excitations forming an emission light signal
• the executable operations may further comprise: controlling a rescanning mirror of the scanning microscope to reflect the emission light signal towards an imaging system and to move a focused emission light signal in accordance with the illumination pattern over the imaging plane of the imaging system; preferably controlling the scanning and the rescanning mirror includes rotating the scanning mirror and the rescanning mirror back and forth over a predetermined first angular amplitude and a second angular amplitude respectively; more preferably the ratio between the first and second angular amplitude unequal to one; even more preferably, the second angular amplitude being selected two times the first angular amplitude.
• the invention may relate to a control module for controlling a scanning microscopy system comprising: a computer adapted to control the scanning
• the computer comprising a computer readable storage medium having at least part of a program embodied therewith; and, a computer readable storage medium having computer readable program code embodied therewith, and a processor, preferably a microprocessor, coupled to the
• control information for controlling a scanning microscope, the control information defining a plurality of different periodic patterns on the basis of one or more pattern parameters, preferably the one or more pattern parameters, including at least one of: a spatial frequency, a periodicity direction, an initial phase; using the control information to control the scanning microscope to expose the sample to multiple illumination patterns, each exposure to an illumination pattern causing one or more optical excitations in the sample, the light originating from said optical
• excitations forming an emission light signal
• the invention may also relate to a computer program product comprising software code portions configured for, when run in the memory of a computer, executing the method steps according to any of process steps described above.
• Fig. 1 depicts a scanning microscopy system according to an embodiment of the invention.
• Fig. 2A-2C depict a process of illuminating a sample with an illumination pattern using a scanning microscopy system according to an embodiment of the invention.
• Fig. 3 depicts illumination patterns for use with a scanning microscopy system according to an embodiment of the invention .
• Fig. 4 depicts the reconstruction of an image using a structured illumination microscopy technique.
• Fig. 5A-5C depicts various illumination light spots for use in a scanning microscopy system according an
• Fig. 6 depicts a scanning microscopy system according to another embodiment of the invention.
• Fig. 7A depicts a calibration of the scanning
• microscopy system in particular of the rescanning mirror with respect to an imaging system.
• Fig. 7B illustrates the principle that the initial phase of a structured illumination light pattern can be determined, even if the spatial frequency of the pattern is so high that it does not have a clear footprint in a captured image .
• Fig. 1 depicts a scanning microscopy system according to an embodiment of the invention.
• the system may comprise a light source 100 for generating illumination light that is directed onto a sample 112 located on a sample holder (not shown) .
• Illumination light 122 may be directed onto a sample using a first optical system 130, which may include one or more refractive and/or reflective optical elements.
• the first optical system may include one or more collimating and/or focusing lenses
• mirrors 102,110 mirrors 104, a set of orthogonal scanning mirrors, schematically depicted by mirror 108 and/or dichroic mirrors 106.
• a lens 102 may be used in order to form a collimated beam of illumination light 122 which may be reflected via a set of mirrors 104,106,108 towards a lens 110 in order to form a focused illumination light spot 126 on the sample.
• the focused illumination light spot may excite local optical excitations, which generate emission light 124.
• the emission light may pass through lens 110 and may be directed via a second optical system 132 comprising one or more
• refractive and/or reflective optical elements to an imaging system 118.
• the second optical system may include a lens 110 for forming a collimated beam of the emission light that is reflected by a first scanning mirror 108.
• the beam of emission light may pass one or more optical elements, e.g. dichroic mirror 106, and reflected by a further second scanning mirror 114, which directs the emission light beams to a focussing lens 116.
• the second scanning mirror may also be referred to as a rescanning mirror.
• the first and/or second scanning mirror may be implemented by a set of
• the focussing lens may focus the emission light beam into focussed emission light spot 128 onto the imaging plane of imaging system 118.
• the imaging system may include one or more image sensors, e.g. one or more CMOS image sensors or one or more CCDs.
• Dichroic mirror 106 may be configured such that it functions as a reflector for illumination light 122 and such that it is transparent for emission light 124 (which typically is of a longer wavelength than illumination light 122) .
• the first scanning mirror 108 which reflects the illumination light towards the sample and the emission light towards the further the second scanning mirror 114 may be configured as a rotatable mirror which may be controlled by a computer system 120.
• the computer system may control the actuators of the scanning mirror so that the light spot can be moved in a 2D plane of the sample. Further, the computer system may control the optics of the system in in order to set a focussing height within the sample. By controlling the focus height different 2D planes in the sample at different focus heights can be illuminated with light patterns. This way, the focused illumination light spot 126 can be moved through the sample 112 in accordance with a predetermined 2D or 3D
• the scanning mirror 108 may be configured to rotate back and forth over a first angular amplitude Al . This way, the mirror can move (“scan") the focused illumination light spot through sample causing continuous local optical illumination in the scanning direction of the light spot.
• Controlling the mirror in three dimensions and controlling the focus of the light spot thus allows writing a 2D or 3D light structure in the sample.
• the computer may comprise one or more processor connected to a memory comprising computer code that, when executed by the one or more processor, may transform a
• predetermined pattern into control information for the optics (the mirrors and the lenses) so that light pattern can be written in the sample.
• the pattern may be defined by a mathematical function including a number of parameters (e.g. frequency, initial phase, amplitude, modulation depth, etc.) which can be easily transformed into control information.
• the control information may control the actuators and optical elements of the system in order to write the pattern into the sample.
• the optical system comprises a feedback system for controlling a position and/or orientation of a mirror, such as the scanning mirror and rescanning mirror.
• the feedback system comprises a device for measuring the position and/or orientation of the mirror and outputs a signal indicative of the measured position and/or orientation towards the computer system.
• the computer system may be configured to, based on the received signal, adjust control signals that are used to control the mirror.
• the mirror is to be controlled such that it makes an angle of 45 degrees to a reference plane. If the device for measuring the orientation then outputs a signal that indicates that the mirror makes an angle of 44 degrees with respect to the reference plane, the computer system may adjust its control signals, e.g. increase or decrease an applied voltage, in order to bring the mirror in the desired position.
• the device for measuring the orientation and/or position of the mirror repeatedly, preferably continuously, measures the orientation and/or position and repeatedly, preferably continuously, outputs the signal towards the computer system in order to allow accurate control of the orientation and/or position of the mirror.
• the one or more processors connected to the memory comprising the computer code may be integrated in a structured light microscopy (SIM) module 132 that is configured to transform a predetermined 2D or 3D pattern in to control information associated with a desired illumination pattern that is used in the SIM process.
• SIM structured light microscopy
• Illumination patterns can be designed in advance using e.g. a software application that may include a drawing program for designing a desired light pattern, wherein the defined pattern represents an exact copy of the illumination light pattern that is written into the sample.
• emission light originating from the moving illumination light spot is formed into a collimated beam 124 of emission light.
• the collimated beam may be reflected via the scanning mirror 108 towards the second scanning mirror 114, a rotatably mounted scanning mirror, which is controlled to reflect the light beam towards the focusing lens 116 in order form a moving emission light spot 128 onto the imaging plane of the imaging system 118.
• the second scanning mirror 114 may be configured to rotate back and forth over a second angular amplitude A2 in order to move (scan) the focused emission light spot over the imaging plane of the image sensor, while the focussed
• illumination light spot is moved (scanned) over the sample by the scanning mirror.
• the actuators and/or electro-motors of the scanning and rescanning mirror may be controlled by a computer system 120 so that the mirrors can be moved
• the frequency of the back and forth rotation of the scanning mirror is identical to the frequency of the back and forth rotation of the rescanning mirror.
• computer system may be configured as a central computer for centrally controlling the actuators and the optics.
• the computer system may be a distributed system wherein different processor may control different parts of the system.
• each scanning mirror may be controlled by a separate processor in order to provide fast low delay control of the scanning mirrors.
• the rescanning mirror is controlled by the computer system to scan associated areas of pixels of the imaging plane of the image sensor so that the pixels are exposed by moving emission lights spots.
• emission light originating from the excited areas is imaged onto the imagining plane.
• the structured light microscopy (SIM) module 132 may control the scanning and rescanning mirror in order to expose the sample multiple times, each time using a different
• illumination pattern typically a periodic illumination pattern, so that during an exposure some parts of the samples are not exposed to the illumination light and other parts are. Examples of illumination patterns are described hereunder in more detail with reference to Fig. 2 and Fig. 3.
• the spatial position of said illumination pattern is spatially shifted and/or rotated with respect to a reference position associated with the sample.
• the exposure of the sample includes
• the focused illumination light signal causes one or more optical excitations in the sample, the light originating from said optical excitations forming an emission light signal.
• each image is associated with one of the multiple exposures.
• the rescanning mirror is controlled to reflect the emission light signal towards the imaging system and to move (scan) a focused emission light signal in accordance with the illumination pattern over the imaging plane of the imaging system.
• the SIM module may construct a high resolution image on the basis of the multiple captured images.
• the second angular amplitude A2 of the rescanning mirror 114 may be larger than the first angular amplitude Al of the scanning mirror 108 so that high- resolution images of the scanned sample area can be obtained.
• the second angular amplitude may be twice the first angular amplitude, so that the
• the ratio between the second and first angular amplitude may be in the range 1-5, particularly 2-5, more particularly 2-4, even more
• the light source 100 may comprise one or more light sources, e.g. one or more lasers, and one or more light filters.
• the light source may be connected to the computer system.
• the one or more light sources and filters may be used to control the wavelengths or bands of wavelengths the illumination light comprises.
• the light source may be configured to generate white light.
• the light source may be configured to generate light consists of two or more
• the light source may generate light of a first wavelength selected from the blue band of the visible spectrum and light of a second wavelength selected from the yellow band of the visible spectrum.
• optical elements of optical system 132 may be configured to control different wavelengths.
• mirror 104 may reflect the illumination light onto dichroic mirror 106 that may be configured to reflect light of a first group of wavelengths, e.g. blue light and yellow light.
• Dichroic mirror 106 may be further configured to pass light of a second group of wavelengths, e.g. red and green light. This way, the illumination light may be reflected by the dichroic mirror onto the scanning mirror.
• a second group of wavelengths e.g. red and green light.
• the illumination light may comprise light of a first wavelength, e.g. yellow light, which may cause a first type of optical excitations in the sample, the first type of excitations generating first emission light.
• a first wavelength e.g. yellow light
• the illumination light may comprise light of a second wavelength, e.g. blue light, which may cause a second type of optical excitations in the sample, the second type of
• first group of wavelengths which may be reflected by dichroic mirror 106, may comprise said first wavelength and said second wavelength.
• sample 122 may be a material, e.g. a biological material, that is imaged using a reflective microscopy technique.
• sample 112 may be an optically active sample.
• the sample may be a material, e.g. a biological material,
• the optically active sample may be non-luminescent optical active material.
• the sample may be a material that can be imaged on the basis of second harmonics generation (SHG) microscopy or third harmonics generation (THG) microscopy.
• the sample may be a material that can be imaged using a Rahman microscopy technique, such as Coherent Anti-Stokes Raman Scattering (CARS) microscopy.
• CARS Coherent Anti-Stokes Raman Scattering
• the sample comprise at least a first type of fluorophores , e.g. red fluorophores , and a second type of fluorophores, e.g. green fluorophores .
• the light of the first wavelength of the illumination light may cause the first fluorophores in the sample to emit the first emission light and the light of the second wavelength may cause the second fluorophores in the sample to emit second emission light.
• the emission light may comprise light that is emitted as a result of photoluminescence, preferably
• a blue light component of the illumination light may cause green fluorophores at the focused illumination light spot to emit green light.
• yellow light component of the illumination light may cause red fluorophores at the focused illumination light spot to emit red light.
• the emission light thus comprises green light and red light.
• the emission light originating from the sample may include optical
• filters such as color filters (not shown) may be positioned in front of the imaging system.
• a color filter may be configured as a bandpass filter that is configured to pass light in a predetermined band of
• the scanning mirror both directs the illumination light signal from the light source 100 towards the sample 112 as well as the emission light signal from the sample 112
• the movement of the emission light signal originating from the sample is neutralized by the movement of the scanning mirror. Due to the movement of the scanning mirror, the path of emission light from scanning mirror to rescanning mirror is static. In other words, the scanning mirror 108 "descans" the emission light 124 and reflects the emission light as a static emission light beam towards the rescanning mirror. Further, the path of the emission light between rescanning mirror and light detector may be dynamic during scanning due to the movement of the rescanning mirror.
• Fig. 2A-2C depict a process of illuminating a sample with an illumination pattern using a scanning microscopy system according to an embodiment of the invention.
• Fig. 2A depicts a top view of a sample 206 wherein a position in the sample may be defined on the basis of a coordinate system, e.g. a 3D Cartesian coordinate system x,y,z.
• the origin of the coordinate system in the sample may be calibrated with respect to a reference position 202 i so that after calibration the coordinate system may be used to
• the reference position may be a position on the sample or associated with the sample, e.g. a position on the sample (holder), e.g. an (optical) marker that is used by the system to position and align the sample within respect to the optical system, in particular the optical axis of the optical system, of the scanning microscope.
• Fig. 2A further depicts part of an illumination pattern 204 i , in this example a periodic arrangement of parallel lines, which is formed by writing the illumination pattern into the sample.
• the computer system may generate or receive control information for controlling the scanning microscope, wherein the control information defines a plurality of different periodic patterns, e.g. a plurality of different stripe patterns, each being defined using one or more pattern parameters, e.g. the number of stripes, the length of the stripes, the spatial frequency, the direction of the periodic tripe pattern (periodicity direction) , an initial phase of the pattern, an illumination light intensity, etc.
• the computer system may use the control information to control the scanning mirror for writing the multiple illumination patterns in the sample.
• Each exposure to an illumination pattern causes one or more optical excitations in the sample and an emission light signal, formed on the basis of the light originating from said optical excitations, may be captured by the imaging system.
• a periodic intensity function identifying high and low illuminated areas in the x- direction of the sample can be identified, wherein the spatial period of the function is identified by a parameter d.
• the intensity pattern in a cross section 200 of the sample in the direction of the periodicity may be described as a sine ⁇ like function rather than a sguare wave function with sharp edges.
• the periodic intensity function of the first exposure in Fig. 2A may be expressed as a sinusoid function 201i and an initial phase cpo :
• I 0 is the averaged intensity
• A is measure of the modulation depth
• 1/d the spatial frequency
• cpo an (initial) phase of the intensity function.
• the initial phase cpo is directly related to the position of the illumination structure in the sample as is illustrated in Fig. 2A.
• the point spread function may cause a certain
• a periodic 2D or 3D illumination pattern can be accurately written into the sample using the control information.
• Pattern parameters including the initial phase and the spatial frequency, defining the different periodic
• the invention thus eliminates or at least substantially reduces the need for a posteriori image analysis in order to determine illumination light pattern parameters that are used by the reconstruction algorithm. This way, light patterns may be used that have a higher spatial frequency than the light patterns that are used in SIM systems known in the prior art.
• the computer system may repeat the process of exposure and image capturing several times wherein each time, the
• the illumination pattern is shifted and/or rotated with respect to the reference position.
• shifted versions of the periodic stripe structure may be written into the sample.
• the shifted illumination patterns of Fig. 2B and 2C may be expressed by similar sine functions 201.2,3 having initial phases (pi and (p2 respectively.
• the spatial shift may be expressed in terms of a spatial phase shift of the periodic intensity function in the x-direction.
• Fig. 3 depicts different illumination patterns for use in an image reconstruction algorithm.
• a first set of light patterns 306i-3 e.g. a set of shifted periodic tripe patterns wherein the stripes may have a predetermined orientation in the 2D plane, e.g. aligned to the y-axis as e.g. described with reference to Fig. 2A-2C.
• the computer system may repeat the exposure of the sample using a second set of light patterns 304i-3 wherein the second set is a rotated version (in this example 45 degrees) of the first set.
• parameters such as an initial phase which may be directly used in the reconstruction algorithm.
• the illumination pattern may have a periodic 2D pattern that has a periodicity in one direction (e.g. such as the pattern
• the illumination pattern may be a periodic 2D pattern that has a periodicity in two directions, e.g. an array of blocks having a periodic in the x and y direction.
• the illumination pattern may be a 3D periodic pattern.
• the sample may be exposed at different focus depths in the (axial) z- direction. Hence, in that case, the illuminated areas are not only separated in the x,y plane shown, but also separated in the axial direction.
• Fig. 4 depicts a process of reconstructing an image on the basis of images obtained by structured illumination microscopy according to an embodiment of the invention.
• the reconstruction process may be executed by computer system of the scanning microscopy system as for example descripted with reference to Fig. 1.
• the illumination pattern may be formed by a periodic illumination pattern which may be
• the Fourier transform may be expressed as follows:
• the optical response of the sample to the structured illumination is a product of the applied light intensity pattern, I (r) , and the fluorophore distribution, S (r) .
• the optical response in reciprocal space is a convolution of the Fourier transform of the illumination pattern 402 and the Fourier transform of the (unknown) fluorophore structure S(k) 404 , which represents an image that needs to be reconstructed.
• the optical response 406 can be represented by:
• the microscope which may comprise one or more lenses, that may be used to capture image data of the optical response may be characterized by an Optical Transfer Function OTF(k) 408 .
• OTF shows that the microscope system
• the dashed vertical lines indicate a cut-off frequency. Frequencies higher than this cut-off frequency will not pass through the example microscope system.
• the Fourier transform of the captured images D(k) 410 is a product (in reciprocal space) of the optical response OR(k) 406 and the OTF(k) 408 .
• This relation may be expressed by the following formula:
• the Fourier transform of the illumination pattern is known. Further, the frequency 1/d and initial phase cpo are pattern parameters which are determined prior to the exposure and can be used directly in the computation. This way, it is mathematically possible to compute the unknown Fourier transform of the fluorophore structure S(k) for a range of frequencies including frequencies
• the computed Fourier transform 414 of the fluorophore structure comprises
• This computed Fourier transform may be used to reconstruct a high resolution image.
• the reconstructed image comprises enhanced resolution in only one direction, namely the direction in which the periodicity exists.
• the above described method has to be repeated using illumination patterns having different periodicity directions, i.e.
• pattern parameters such as the initial phase and the spatial frequencies defining the
• Fig. 5A-5C depicts various illumination light spots for use in a scanning microscopy system according an
• illumination light spots i.e. illumination light spots with different light intensity profiles
• the darker areas are associated with high light intensities and the lighter areas with low light intensities.
• the illumination light spots move in the
• the dashed lines indicate one or more areas of the plurality of areas, i.e. the illuminated areas on the sample, after the illumination light spot has moved over the sample in the direction as specified by the arrow.
• FIG. 5A shows an illumination light spot that has a simple Gaussian shaped light intensity profile.
• FIG. 5B shows an illumination light spot that comprises a predetermined light intensity pattern.
• the illumination source of the scanning microscopy system and the optics associated with the illumination source may be adapted to generate an illumination light spot 5042 comprising at least two intensity maxima wherein d defines the distance between the maxima.
• d defines the distance between the maxima.
• the illumination light spot may be shaped, e.g. oval instead of circular so that the width of the spot in the scanning direction is larger than the width of the spot in the
• the orientation of the shaped light spot has to be changed when changing the scanning direction so that the width of the spot perpendicular to the scanning direction is always the smallest width. This way, a fine resolution of the
• Fig. 5C depicts a further embodiment wherein the illumination light spot comprises a plurality of maxima so that a plurality of lines are generated when moving the illumination light sport in predetermined directions.
• the maxima may be arranged in a square configuration, so that when moving the light sport in a first direction a plurality of lines are simulations written onto the sample and when moving the light sport in a second direction a plurality of lines are simultaneously written onto the sample.
• the plurality of maxima may be arranged in a configuration having three axis of rotation symmetry, i.e. the x-axis, y-axis and an 45 degree axis.
• the scanning direction of the illumination light spot is parallel to one of the directions of axis of symmetry, the same striped pattern can be generated.
• the shape of the illumination light spot and the number of maxima in the illumination light spot may be any shape.
• optical element that can shape the phase-profile and/or intensity profile of the wave front of the excitation laser beam, such as a deformable mirror of a spatial light modulator.
• Such optical element is controlled by a computer with some control software.
• Fig. 6 depicts a scanning microscopy system according to another embodiment of the invention.
• Fig. 6 depicts a scanning microscopy system similar to the system described with reference to Fig. 1, comprising a light source, a sample 612, an imaging system 618 and optical elements including a mirror 604, dichroic mirror 606, scanning mirror 608, focusing lens 610, rescanning mirror 614 and focusing lens 616.
• the system in Fig. 6 comprises an optical arrangement 621 for enabling confocal measurements wherein the optical arrangement may comprise a pinhole 623.
• the scanning microscopy may be
• the system may be configured to arranged to focus emission light from the plane of interest P onto a plane 623 P' .
• the system may comprise one or more optical elements, e.g. lenses 620 and 622, arranged to focus emission light from the plane of interest P onto plane P' .
• Plane P' thus may be a confocal conjugate plane of plane of interest P.
• the scanning microscopy system may comprise a point spread function (PSF) module 603 that is configured to modify the illumination light spot by modifying the shape of the illumination light spot and/or introducing a predetermined arrangement of light intensity maxima in the illumination light spot as e.g. described with reference to Fig. 5A-5C.
• PSF point spread function
• Fig. 7A illustrates an exemplary calibration of the optical system.
• the calibration comprises exposing a reference sample 754a to a reference light pattern 704a and capturing an image 758a, for example using methods described above.
• the dashed outline 750 indicates a first state of the optical system.
• the rescanning mirror 714 is oriented at a first angle with respect to a reference plane, e.g. the imaging plane of the imaging system, and the scanning mirror 708 is oriented at a second angle with respect to the reference plane.
• a reference plane e.g. the imaging plane of the imaging system
• the scanning mirror 708 is oriented at a second angle with respect to the reference plane.
• the optical system is in the second state 752
• at least one of the scanning mirror and rescanning mirror is oriented differently with respect to the orientations in the first state.
• the rescanning mirror 714 is oriented at a third angle with respect to the reference plane that is different from the first angle and the scanning mirror 708 is positioned at a fourth angle with respect to the reference plane that is different from the second angle.
• a focused illumination light spot is incident on the sample 754a at position 761.
• the focused illumination light spot is incident on the sample at position 762.
• the reference sample 754a in this example has a substantially homogeneous fluorescent layer, preferably of thickness less than 200 nm, more preferably less than 100 nm, which means that a focused illumination light spot positioned anywhere on the reference sample will cause optical
• a focused emission light spot may be formed on the basis of this light emission, which light spot may be scanned over an imaging plane 756 as
• Reference numeral 756 indicates an imaging plane of an imaging system, for example of a CCD camera.
• the imaging plane 756 comprises a plurality of pixels two of which are indicated by 775a and 775b respectively.
• a focused emission light spot is scanned over the imaging plane 756 in accordance with the pattern indicated by 760a.
• the illumination light spot is at position 761 on the sample 754a, the emission light spot on the imaging plane 756 is at
• the emission light spot on the imaging plane 756 is at position 764.
• the focused emission light spot is incident on positions 762 and 764 on the imaging plane 756 when the optical system is in the first state 750 respectively second state 752.
• Image 758a is the image captured by the imaging system during the calibration procedure as a result of the exposure of reference sample 754a to the reference
• the pixels 755 of the imaging plane 756 may be one-to-one associated with image pixels of image 758a. Each image pixel may thus indicate how many photons were received by its associated pixel 755 of the imaging plane 756 during a certain time period.
• the three vertical paths 770 indicate bright regions in the image 758a as a result of exposing the
• reference sample 756a to the reference illumination light pattern 704a and scanning the emission light spot over the imaging plane 756 in accordance with pattern 760a.
• the bright regions 770 comprise a region 766 that represents illuminated part 761 of the sample 754a and
• the calibration method may then comprise storing calibration information by storing the first state of the optical system, for example the respective angles that the scanning and/or rescanning mirrors make with respect to a reference plane, in association with region 766 of image 758a and/or in association with position 763 on the imaging plane 756 and/or in association with one or more pixels at position 763 of the imaging plane.
• the calibration method may comprise storing calibration information by storing the second state of the optical system, for example the respective angles that the scanning and/or rescanning mirror make with a reference plane, in association with region 768 of image 758a and/or in association with position 764 on the imaging plane 756 and/or in association with one or more pixels at position 764 of the imaging plane.
• Storing a state may be performed by storing control information for bringing the optical system in that state.
• storing a state may comprise storing a voltage that is to be applied to actuators of a scanning and/or rescanning mirror for causing the scanning and/or rescanning mirror to orient such that the optical system adopts the state .
• the emission light pattern 770 is discernible in image 758a. Based on image 758a alone, the exact regions in the image can be determined that represent parts of the sample that have been exposed to the illumination pattern 704a. Phase retrieval algorithms known in the art may be executed for finding the exact position of the pattern 770 in the image 758a.
• a state of the optical system may thus, in one embodiment, be defined by the respective orientations of the scanning and rescanning mirror.
• the first state is not arbitrarily chosen, but preferably is predetermined based on pre-calibrations of the optical system.
• the scanning mirror is preferably oriented such that the reference illumination pattern 704a, for which the optical system may have to adopt the first and second state, falls properly onto the reference sample 754a.
• the scanning mirror may have to be pre-calibrated with respect to a sample holder according to methods known in the art.
• pre-calibrating the scanning mirror with respect to the sample holder may comprise imaging a test sample positioned in the sample holder, wherein the test sample comprises a known fluorescent structure, such as fluorescent lines that are spaced 1 micrometer apart over a length of 1 mm.
• Another pre-calibration may relate to calibrating the scanning mirror and rescanning mirror with respect to each other, which may be performed as follows. First, a sample comprising a number of fluorescent beads is positioned in the microscope, e.g. in or on a sample holder. Then, while the excitation laser of the microscope is shut off and any pinhole is removed, an external light source illuminates the entire sample and thus excites all beads. The fluorescent emission light of the beads travels from scanning mirror to rescanning mirror and is incident on the imaging system and an image of the beads appears on a display connected to the imaging system. The scanning and/or rescanning mirror may be moved, e.g. rotated, such that the emission light is properly
• each possible orientation of the scanning mirror may be associated one to one with an orientation of the rescanning mirror .
• a first angular amplitude of rotation of the scanning mirror is compensated by a second angular amplitude of rotation of the rescanning mirror.
• the first and second angular amplitude need not be the same depending on the optics between the scanning and rescanning mirror. After the first and second amplitude have been found, the second
• amplitude may be doubled when using the microscope to capture an image of a sample, because this may enable to enhance the resolution of the captured image as described in Biomed Opt. Express; 2013 Nov 1; 4(11) : 2644-2656.
• FIG. 7B illustrates capturing one of the multiple images during an embodiment of the method for forming a high resolution image of a sample 754b.
• the sample 754b comprises coarse structures 772, such as structure 772a and 772b.
• sample 754b comprises fine structures 774, such as 774a and 774b.
• the embodiment comprises a processor controlling a scanning microscope to expose the sample 754b to illumination pattern 704b.
• the illumination pattern 704b in this example is finer than the reference illumination pattern 704a as shown.
• exposing the sample to illumination pattern 704b comprises moving a focused illumination light spot over the sample 754b in accordance with the illumination pattern 704b. While exposing the sample to the illumination pattern 704b, the optical system may be understood to be controlled to adopt the first state 750 and optionally the second state 752, in this example identical to the first and second state mentioned with reference to Fig. 7A, so that the focused illumination light spot is incident on the sample 754b at position 775.
• the focused emission light spot may be
• the image 758b of the sample 754b shows the coarse structures 772 in the sample 754b.
• Image 758b does not show the fine structures 774 of the sample, because these fine structures are associated with high frequencies that are attenuated too much by the optical transfer function of the optical system as schematically illustrated in Fig. 4,
• the illumination pattern 704b is so fine, that the image 758b does not have a clear enough
• the intensity minima and maxima of the illumination pattern 704b cannot be distinguished in the image 758b, which for example prevents a phase retrieval algorithm known in the art to determine the exact regions in the image that represent parts of a sample exposed to
• the processor can determine that region 766 indicated in image 758b represents a part of the sample 754b that was exposed to illumination.
• region 768 in image 758b may likewise be determined to represent a part of the sample that is exposed to illumination light based on the second state 752 being associated with region 768 and the optical system having adopted the second state while exposing the sample 754b to illumination pattern 704b.
• the method comprises determining a region 780 in an image as representing a part of the sample exposed to illumination on the basis of the calibration information.
• the region 780 may be associated with a further state of the optical system. It is thus not required that the optical system adopts an identical state during exposing the sample to illumination pattern 704b and during the calibration procedure described with reference to FIG. 7A. Based on the calibration data further links can be made between states of the optical system and respective regions in captured images.
• the method further comprises capturing further images associated with respective further illumination patterns to which the sample 754b is exposed which allows to form a high-resolution image on the basis of the obtained images using an image reconstruction algorithm.
• Input to the image reconstruction algorithm is for example that region 766 in image 758b represents part of the sample that was exposed to illumination, which allows
• sample 754b may be used as reference samp1e 754a.

Landscapes

• Physics & Mathematics (AREA)
• Engineering & Computer Science (AREA)
• Chemical & Material Sciences (AREA)
• Analytical Chemistry (AREA)
• General Physics & Mathematics (AREA)
• Optics & Photonics (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Multimedia (AREA)
• General Engineering & Computer Science (AREA)
• Microscoopes, Condenser (AREA)

Applications Claiming Priority (2)

Publications (1)

Family

ID=58501785

Family Applications (1)

Country Status (3)

Families Citing this family (3)

Family Cites Families (2)

• 2017
  • 2017-02-16 NL NL2018386A patent/NL2018386B1/en active
• 2018
  • 2018-02-16 EP EP18710930.1A patent/EP3583459A1/de active Pending
  • 2018-02-16 WO PCT/NL2018/050107 patent/WO2018151599A1/en unknown

Also Published As

Similar Documents

Legal Events