JP2021513095A - Methods and systems for localized microscopy - Google Patents

Abstract

Implementations of the present invention provide methods and systems for processing microscopic images to allow localization under high densities, thus achieving higher spatial resolution than otherwise. This is achieved by taking advantage of the temporal redundancy of the image data obtained from adjacent illuminants. These adjacent illuminants would otherwise be separated as a single illuminant that simultaneously emits or fluoresces, but at slightly different (but potentially overlapping) times. Because it emits light or fluoresces, it can be time filtered by different filters with different time bandwidths, thus separating the two illuminants. In fact, the different time filters have different time constants that work together to effectively emphasize the different emission times or fluorescence times of the two emitters, thereby separating the two adjacent emitters. Can be separated into.

Description

The practice of the present invention reduces the density of observed emitters / fluorophores in individual image frames by sequentially applying sliding time filters on multiple time scales. The present invention relates to a method and a system for processing an image sequence in localized microscopy. Due to the reduced density, high-density and low-density data analysis methods are sequentially applied without generating image artifacts that would normally occur when these methods are applied to data containing high-density illuminants. It is possible to apply to these data.

Localized microscopy is a technique in optical microscopy for obtaining image resolution that exceeds the diffraction limit, which is generally called super-resolution. This involves chemically labeling the structure of the sample with fluorescent molecules that exhibit intermittent or fluorescent "blinking". Fluorescence intermittentness can be controlled by adding a chemical substance and by irradiating with light having an appropriate intensity (intensity) and wavelength. Means for fluorescently labeling the structure of a sample and controlling the intermittent emission of fluorescence are roughly classified into two types. Stochastic Optical Reconstruction Microscopy (STORM) labels a sample with a fluorescent organic dye molecule. Fluorescence intermittentness is usually promoted by the addition of an oxygen scavenging enzyme and a chemical reducing agent. Photoactivation localization microscopy (PALM) uses a fluorescent protein for structural labeling, which is non-emission by applying short wavelength visible / near-ultraviolet light. It is possible to switch from one state to another, or from one state to another with a longer wavelength (ultraviolet light can also reactivate the illuminant / phosphor in STORM).

When an image of a sample is taken into a typical localized microscopy apparatus, it will consist of a plurality of fluorescent "spots" whose size is close to the diffraction limit. The size of these spots defines the device's point image distribution function (or point spread function, Point Spread Function, PSF). Each spot is generated by a single illuminant (emitter) / phosphor (fluorofore) that has been naturally or intentionally switched to the appropriate luminescent state. If these fluorescent spots are sufficiently separated so that they do not overlap each other, it can be arguably assumed that only one active illuminant / phosphor contributes to the spot. The position of the illuminant can be identified from the fluorescence emission with an accuracy much higher than the size of the fluorescence spot (usually an accuracy of 10 nm to 30 nm). This is a process known as localization (localization), which is always performed by algorithms implemented in computer software. Once a sufficient number of illuminants / phosphors have been successfully localized, the structure of the sample can be estimated in high resolution from these localized locations. The luminescent image of the illuminant / phosphor is usually acquired as a long series of continuous image frames using a high sensitivity (sCMOS / EMCCD) camera attached to a conventional wide-field fluorescence microscope. In each sequential frame, some of the active illuminants / phosphors switch from the luminescent state and some of the previously non-emitting illuminants switch to the luminescent state. The requirement that the active illuminants / phosphors must be sufficiently separated will usually limit the number of active illuminants / phosphors in each frame to a relatively small number. Tens of thousands of image frames are required to accurately reconstruct the structure of the sample. The speed limitation of the camera used means that it takes a few minutes to acquire the image, which is a time-consuming process for many dynamic changes to the structure under study, such as changes that are constantly seen in living cell samples. It's too long to catch.

Examples of prior art described and published in the above summary include: (References in STORM) "Sub-diffraction-limited imaging by stochastic optical re-construction microscope", Rust, M. et al. , Bates, M. et al. , And Zhuang, X. et al. Written by Nat. Methods, Vol. 3, pp. 793-769, 2006; (PALM Bibliography) "Imaging intracellular fluororescent protein at nanometer resolution", Betzig, E. et al. Et al., Science, Vol. 313, pp. 1642-1645, 2006; and (PALM Reference Patent Documents) "Optical Microscopic with Phototransformal Optical Labels", WO 2006/127592, Hess, Halld.

The acquisition speed can be significantly increased by increasing the number of active illuminants / phosphors in each image frame and reducing the number of frames required for accurate reconstruction. However, localization methods / algorithms must be able to address the substantial overlap of fluorescence emission patterns (spots) of individual emitters. A form of experiment in which the typical distance between active illuminants is less than PSF is called high density localized microscopy. When this form is reached, standard monoluminous algorithms such as single Gaussian fitting will fail to detect and / or accurately locate each illuminant. In addition, the close and separated illuminant / phosphor fitting positions include a bias towards the center of each other. This leads to substantial artifacts such as artificial sharpening, false clustering, and lack of structure in the reconstructed super-resolution image. Artificial sharpening is particularly problematic because it gives the false impression of high resolution, even when measured with metrics such as Fourier Ring Correlation (FRC). Currently, there is no way to detect the presence of these artifacts in localized microscopy reconstruction. Little is known about the ground truth structure of the samples tested, so it is easy to mistake these artifacts for real structures.

Representative references to the above further prior art include: (Artifacts references) "Artifacts in single-molecule localization microscopy", Burgert, A. et al. , Letschert, S.A. , Doose, S.A. And Sauer, M. et al. Written by Histochemistry and Cell Biology, Vol. 144, pp. 123-131, 2015; (References to Artifacts) "Local Dimensionality determines imaging, speeded in localization, Microscopy. E. et al., Nature Communications, Vol. 8, p. 13558, 2017; and (FRC Reference) "Measuring image resolution in optical nanoscopy", Nieuwenhuizen, R. et al. P. J. Others, Nature Methods, Volume 10, pp. 557-562, 2013

Several approaches have been developed to address the overlapping situation of illuminant PSFs, but nonetheless, if the illuminant densities are high enough, all of them still produce image artifacts. These approaches are briefly summarized below.

i) Multi-Emitter fitting method
This method is an extension of the standard single illuminant-single frame fitting method used for low density illuminant data. In this method, the local maximum value of the PSF scale image is identified and a small portion (section) of the image surrounding that maximum value is selected (similar to single light emitter fitting). For one or more illuminants, a typical PSF is fitted to this portion. Usually, the number of contributing illuminants increases until no statistically significant improvement in fitting is observed. Usually, a regularization term is introduced to limit the number of fitted phosphors and / or the constraints imposed on their intensities. This method is not too small (about 250 nm) between the illuminants, and the density is still low enough to observe a well-defined local maximum, and the number of overlapping phosphors is reasonable. (Fairly) When it is low, it works effectively. However, if any of these conditions are not met, it tends to fail. Prior arts for compound illuminant fitting include, for example: (ME fitting references) "Simultaneus multiple-emitter fitting for single molecule super-resolution imaging", Huang, F. et al. , Schwartz, S.A. , Byrs, J. et al. And Lidke, K. et al. Written by Biomedical Optics Express, Vol. 2, pp. 1377-1393, 2011; and (ME Fitting Reference) "DAOSTORM: an algorithm for high-density super-resolution microscope", H. J. , Upoff, S.A. And Kapanidis, A. et al. N. Written by Nature Methods, Volume 8, pp. 279-280, 2011.

ii) Compressed sensing method
Compressed sensing is a well-established signal processing method that covers a wide range of fields and applications. This method is optimal because under certain conditions it can substantially restore an unsampled signal if it is sufficiently sparse (contains almost zero). It is a conversion method. In the case of super-resolution microscopy, this method means performing a higher resolution transformation for each image frame with more and smaller pixels. Then, all these converted frames are summed up to generate a final super-resolution image. Certain conditions must be met in order for the optimization to be performed. The structure of the sample must be sparse, that is, most samples must not contain a labeled structure. This condition may apply globally to biological structures such as microtubules, but may not necessarily apply locally. Also, the PSF of the device must be accurately known and constant everywhere. This is unlikely to be the case, for example, as some focal drift or aberration is almost always present. The background change should also be small relative to the size of the image being analyzed.

This method is significantly superior to single illuminant fitting in that it identifies individual illuminants when the illuminants are dense, while the typical sensation of a phosphor is up to half the full width of the PSF. Below the value, the accuracy drops significantly. Compressed sensing methods use several mathematical methods to perform optimization, and these methods are known to generate artifacts under certain conditions. It is not yet known how these are applied to super-resolution microscopy.

Prior arts for compressed sensing microscopy include, for example: (References for Compressed Sensing Microscopy) "Faster STORM compressed sensing", Zhu, L, Zhang, W. et al. , Elnatan, D. et al. , And Huang, B. et al. Written by Nature Methods, Vol. 9, pp. 721-723, 2012.

iii) SOFI: Super-resolution optical variation imaging method In this method, intensity variation over the entire image sequence is analyzed. Calculate self or mutual cumulants in a time trace of pixel brightness at a specific order. The fluorescee density is high where the variability is clustered, so this method does not associate individual variability with individual fluorophores. The map of the calculated values forms a super-resolution image. The artifacts known in this method tend to exaggerate the brightness of the sample region above the average relative intensity, which is adversely affected by the "blinking" properties of each fluorophore. .. As a result, it may not be possible to detect a structure with low brightness in the super-resolution image. In order to obtain a resolution significantly higher than the diffraction limit, the cumulant must be calculated to a higher order, which greatly exacerbates this problem.

The prior art of SOFI may include, for example: (references of SOFI) "Fast, background-free, 3d Super-resolution optical optication imaging (SOFI)", Dertinger, T. et al. , Colliera, R. et al. , Iyera, G. et al. , Weissa, S.A. , And Enderlein, J. et al. Written by Proc. Natl. Acad. Sci. , 106, pages 22287-22292, 2009.

iv) 3B: Blinking and fading in the Bayesian method (Bayesian blinking and bleaching)
The approach adopted here is an attempt to find a model structure of a sample that fits the entire image sequence showing the behavior of optical switching (“flashing”) and photobleaching of the phosphor. Starting with an initial guess, the position of the phosphor is repeatedly adjusted until no further improvement is observed. This method can obtain high resolution from a reasonably small number of frames, but if the spacing between the illuminants is small, it is affected by the sharpening artifacts similar to the compound illuminant fitting. Also, the amount of calculation is very large, and the calculation time is much longer than any other method. Therefore, application is limited to very small images unless large computational resources are available.

Prior art can include, for example, the following: (References in 3B) "Bayesian localization microscopy microscopy revolutions nanoscale podosome dynamics", Cox, S. et al. Other work, Nature Methods, Vol. 9, pp. 195-200, 2012.

v) SRRF: Super-resolution radial fractionations
SRRF can be regarded as a hybrid system that combines localization of the illuminant and higher-order time correlation (SOFI, etc.). In the first part, a local radial symmetry sub-pixel map within the image is generated from the spatial gradient of pixel brightness. Where the illuminant is located, the degree of local radial symmetry should be high. One of these "radiality maps" is generated for each frame in the series. After that, the entire series of "radial maps" is analyzed in the same way as SOFI, but this time, the time correlation in the time trace of the sub-pixels of each "radial map" is calculated. Then, these values generate a final super-resolution image. This method has both advantages and disadvantages for localization and time analysis. Useful resolution gains can be obtained at both stages, but with the brighter parts of the structure seen in SOFI, they produce artificially sharpened artifacts for adjacent phosphors, similar to other fitting methods. The contrast with the darker part / missing part is also exaggerated.

Prior arts of SRRF include, for example: (References of SRRF) "Fast live-cell conventional fluorophore nanoscopy with ImageJ Thursday super-resolution radiation, Gustafsson. Other work, Nature Communications, Volume 7, page 12471, 2016.

vi) BALM: Bleaching / blinking assisted localization microscope
This method does not look at the entire image sequence to get time information that helps increase the resolution, but simply subtracts the other frame (adjacent pair) from one frame to get two frames at a time. It's just a comparison. This method targets fluorescent labels that do not show significant intensity switching behavior or do not use optical switching buffers. Initially, most phosphors are in a luminescent state. Over time, the switch turns off or fades. Fluorescent materials that switch "off" or photobleach in a particular image frame can be detected by subtracting subsequent frames from the current frame. The localization algorithm is then applied to this "difference" frame. By subtracting the preceding frame, the more rare "switch-on" event can be isolated.

However, this method has some drawbacks compared to conventional localized microscopy and the other high density methods described above. Most importantly, this method is arguably not a high density method. Since the illuminant density of each difference frame is limited by the ability of the localization algorithm to resolve the illuminant proximity, similar numbers can be used to generate high fidelity super-resolution reconstructions. Frame is required. Overlapping illuminants that are "off" (or "on") are not separated in the same frame. Similarly, if one illuminant is switched "off" and nearby illuminants are switched "on" within the same frame, they will be partially ambiguous to each other in the "difference" frame and are uncertain. It leads to the localization. If the phosphors only switch / fade slowly, most of their emitted photons are ignored from the difference frame, leading to inaccurate localization (if not biased). When most phosphors emit light, the brightness values of individual pixels are very high. Photon shot noise and camera read noise are also higher than the difference signal. Therefore, the signal-to-noise ratio is much lower than that of the conventional localized microscopy, which leads to a decrease in localization accuracy and a low resolution.

Prior arts of BALM include, for example: (References of BALM) "Blinking / blinking assisted localization microscopic for super-resolution imaging standard fluorescence" standard fluorescence. T. , Sengupta, P. et al. , Dai, Y. et al. , Lippincott-Schwartz, J. Mol. And Kachar, B. et al. Written by PNAS, Vol. 108, pp. 21081 to 21086, 2011.

vi) Multicolor detection
All of the methods mentioned above can be used in combination with the multicolor detection method. In this variation of localized microscopy, two or more different structures in a sample are labeled with fluorescent molecules of different emission wavelengths (colors). The equipment included is the same as for a conventional localized microscope, but an optical filter is added to the optical path of the collected fluorescence emission. The passband of the optical filter is selected to separate the emission wavelengths of the two fluorescent marker molecules. The filtered fluorescence then becomes an image on separate cameras, or an image on separate areas of the sensor of a single camera. Therefore, a plurality of image sequences corresponding to each fluorescence marker can be generated, and a plurality of structures in the sample can be imaged at the same time. However, it can be difficult to manipulate the proper "blinkability" of multiple fluorescent markers, and this method can also be affected by chromatic aberration. Prior arts for multicolor localized microscopy include, for example: (References for Multicolor Methods), "Subdifraction-Resolution Fluorescent Imaging With Conventional Fluorescent Probes", Heilemann, M. et al. , Van de Linde, S.A. , Schuttpelz, M. et al. , Kasper, R. et al. , Seefeld, B. et al. , Mukherjee, A.M. , Tinnefeld, P. et al. And Sauer, M. et al. Written by Angew Chem Int Ed Engl. , Vol. 47, No. 33, pages 6l 72-6, 2008.

Implementations of the present invention provide methods and systems for processing microscopic images that allow localization under high densities and thus achieve higher spatial resolution than otherwise. This is achieved by taking advantage of the temporal redundancy of the image data obtained from the adjacent illuminants, which would otherwise emit or fluoresce at the same time. Separated as a single illuminant, but emits or fluoresces at slightly different (but may overlap) times, so it should be time filtered by different filters with different time bandwidths. Therefore, the two illuminants can be separated. In essence, the different time filters have different time constants that work together to effectively emphasize the different emission times or fluorescence times of the two illuminants, thereby causing the two to be in close proximity. The illuminant can be separated separately.

In view of the above, one aspect provides a method for localized microscopy. That is: a step of receiving a time series of microscopic image frames captured by a microscope, wherein the image frame is an image containing a illuminant or phosphor in it; with a step; at least one time filter. By temporally filtering each set of pixel values at each same pixel position in the frame series, it is temporally filtered for each same pixel position in the frame. Steps to obtain at least one set of pixel values; provide at least one set of temporally filtered pixel values as input to the localization algorithm and rely on it to localize the illuminant or phosphor. Includes steps and;

In one embodiment, this temporal filtering is at least two having different temporal properties in order to obtain at least two sets of temporally filtered pixel values for each identical pixel position in the frame. At least two sets of temporally filtered pixel values, including the step of using different temporal filters, are provided to the localization algorithm to allow localization of the illuminant or phosphor.

Further features and aspects will become apparent from the appended claims.

Further features and advantages of the present invention will become apparent from the following description of the practice of the present invention, as well as the accompanying drawings, provided by way of example only. Here, similar reference numerals indicate similar parts.

It is a flow diagram of a typical embodiment of an algorithm. It is a figure which shows the localized microscope apparatus which concerns on one embodiment of this invention. It is a figure of the frame series before and after the treatment which concerns on one execution, and shows the density of the light emitting body which was reduced. It is a figure which showed an example of the filter set which uses a Haar wavelet.

Each practice of the present invention includes methods and systems for processing a series of images obtained using localized microscopy to reduce the number of active illuminants / phosphors in each image. This technique is applied to the image sequence prior to any localization method / algorithm, and the output of this technique, which is the subject of the invention, is a much longer image sequence and the active illuminant in each image frame / The number of phosphors is decreasing. The selected localization method / algorithm is then applied to the output image sequence to locate the individual illuminants / phosphors and generate a super-resolution image. Another practice of the present invention would be to combine multiple temporal filtering methods with localization methods / algorithms into a single method / algorithm. The filtering process separates the emission of overlapping emitters / phosphors based on the number of adjacent frames appearing in the emission state. More precisely, it is based on the number of adjacent image frames that maintain a reasonably constant intensity before switching to a brighter or darker luminescent or non-luminous state.

A set of filters is any set that constitutes a comparison of pixel values of adjacent / adjacent frames on various time scales. Each filter will have a characteristic time scale that represents the time (related to the number of frames) between the first and last image frames it compares. Each filter then separates the intensity fluctuation event / switching event of the illuminant on its characteristic time scale from all other events. The time scale range used will reflect the time scale range of illuminant variation / switching to capture most of these events. The upper limit of the filter time scale is generally much shorter than the length of the image sequence, but usually there will be no variation in the illuminant on such a long time scale.

This filter is applied to a time trace of a series of pixel brightness composed of the input image sequence. The luminance time trace is generated by sequentially collating (connecting) the values of specific pixels x and y in all the image frames in this series. A filter with a time scale with a particular characteristic is applied to a particular point in the pixel brightness time trace and the recorded output value. The filter is then applied to multiple points in the time trace (usually multiple, but not necessarily all points), and each resulting value is added to the previous value to generate a time trace of the output pixel brightness. To do. This step is repeated for each brightness trace of each pixel in the original image sequence to generate an equal number of time traces of output pixel brightness. These output luminance traces are reconstructed at the original pixel positions x and y to create an output image sequence. This process is then repeated for the next filter in the set and all subsequent filters, and the image sequence for each output is added to the previous one, much longer than the original sequence. A total output sequence is generated.

The order in which the filters are applied to different points in the pixel brightness time trace is not important unless a temporal evolution of the structure is observed and it is maintained for all such traces. The order in which each filter is applied is also not important. Alternatively, each filter can be applied sequentially to a single input luminance time trace and its output added to a much longer output time trace. As long as the order in which the filters are applied is consistent, these long output traces are then combined into the final output image series as described above.

When the brightness of the light emitter fluctuates, or when the light emission state of the light emitter changes, the time traces of all the pixels within the PSF width at that position indicate the brightness change (variation) at that time. In the absence of other phosphors with overlapping initial switching brightness within the same image frame, the relative magnitude of these variations between adjacent pixels will reflect the PSF of the device. However, when two or more adjacent illuminants in the same image frame switch brightness, some pixels within the PSF range of the illuminant will experience a change in brightness due to the plurality of illuminants in the time trace. On the other hand, the other pixels show only the brightness due to only a single illuminant. However, in any of the other illuminants that overlap the PSF of a single illuminant, it is extremely rare for both front and rear brightness switching to occur in the same relative quantity and in the same image frame. .. Therefore, the contribution of each illuminant to the variation in brightness of individual pixels from frame to frame can be separated based on the number of image frames in which the variation persists. This is achieved by time filtering with multiple time scales. By comparing the brightness values of the pixels over several frames, the output of each filter is an image sequence containing only a subset of the brightness variation, the time scale of which depends on the characteristic time scale of the applied filter. To. Again, when individual emitters switch brightness, all filtered pixel traces within that PSF show an output that matches the value representing the shape of the PSF, but this time it is characteristic of the applied filter. It is limited to the illuminant that switches in the time scale set by the time scale. Here, when signal noise from light emitters of different time scales passes through the filter, the same spatial match is not shared and the intensities of adjacent pixels do not reproduce the PSF of the device. Localization algorithms are designed to identify intensity distributions according to the device's PSF, eliminating noise while locating them.

In other words, the output of the short time scale filter is an image sequence containing fluorescence "spots" of only the emitter / phosphor that were in the emission state during a particular small number of image frames. On the other hand, the output of the long time scale filter includes only the illuminant / phosphor that was emitting light between a large number of frames. The number of illuminants in each frame (and the overlap of illuminant densities and PSFs) is significantly reduced, and localization algorithms can more accurately locate illuminants. By combining the time scale results of multiple filters, almost all illuminants / phosphors appear in at least one image frame. This allows for a sufficient number of localizations to generate high resolution reconstructions without significant image artifacts.

Time filtering by multiple time scales is further used as an alternative / supplement to multicolor imaging. The output of the shorter time scale and the longer time scale filter can be combined into a "high pass" and "low pass" pair (or bandpass if divided into three or more groups). ). If the sample was labeled with one "fast / good" flashing and one "slow / inferior" flashing, each filter, even if they had the same emission wavelength. The pair will primarily contain luminescence from either one or the other of the two fluorophores. Therefore, the disadvantages of multicolor imaging mentioned above are avoided.

Further features and advantages will become apparent from the detailed description below.

Examples of the present invention will be described herein with reference to each figure. In particular, FIG. 2 captures an image that will be processed according to the techniques described herein to allow high resolution localization of the illuminant / phosphor in the imaged sample. An example of a microscope device that can be used is shown.

More specifically, FIG. 2 shows an example of an apparatus commonly used in the acquisition of localized microscopy image sequences according to one embodiment of the present invention. The fluorescently labeled sample (204) of interest is held by the sample stage (202) in a fixed / adjustable relative position with respect to the high numerical aperture objective lens (206). One or more excitation sources (220) generally consist of one or more excitation wavelengths and one or more activation wavelengths, both wavelengths along a common optical path by the beam-coupled optics (222). Is oriented. The light source may be a laser or another collimated, collimated, collimated, high intensity light source. In addition, beam intensity distribution and collimation can be modified using beam conditioning optics such as beam expanding telescope (218). The light source is then reflected onto the sample (204) through the objective lens (206) by a dichroic mirror (major dichroic 208). Coupling and directing the light source to the position of the major dichroic (208) can also be achieved partially or wholly by fiber optics. Beam directional / adjustable optics usually provide total internal reflection fluorescence (TIRF) of excitation light at the interface of the sample by providing some means of altering the beam path through the objective lens.

Fluorescent light from the individual illuminants / phosphors is collected by the objective lens and directed by the optical system (216) to pass through the main dichroic and through the tube lens (210). This focuses the light on the sensor of the digital camera (212) that forms the image. The camera needs to be sensitive enough to detect the faint levels of light it contains, and it needs to be fast enough to capture multiple (about 100) images per second. Therefore, a CMOS (sCMOS) for scientific measurement or an electron multiplier CCD (EMCDD) camera is usually used. The output data stream is recorded on the computer (214).

The computer (214) is provided with a data storage medium 224 such as a hard disk, a flash drive, and a memory card, and is processed on the data storage medium 224 in the form of pixel data obtained from an image captured by a microscope device. Data, processing output in the form of processed image, and an executable program that controls a computer at the time of execution to execute necessary processing for the input image and acquire the output image. Can store a general control program (226), a Harr Wavelet Kernel (HWK) calculation program (228), and a localization program (230). The general control program (226) controls the entire process and calls the HWK calculation program and the localization program as needed. The HWK calculator (228) receives each input stream of pixel data associated with each pixel of the captured microscopic image and applies the Haar wavelet function to them as described below for each pixel. On the other hand, an output stream of pixel data is generated. Here, the pixel data filtered by each Haar wavelet function is output, and each output is obtained by applying it to each of the Haar wavelet functions of different orders (for example, 1st to 4th order). And their outputs are concatenated or combined to form a pixel stream that is output pixel by pixel. The output pixel stream contains both positive and negative values as the brightness of the input pixel stream increases or decreases. To process these, either take the absolute value or double the size of the output pixel stream to separate the positive and negative values into two consecutive output pixel streams and then absolute. Add a value. In such cases, all missing values are filled with zeros. The third option is to add a large fixed offset value to all output pixel values so that the output image sequence has no negative pixel values (eg, half the dynamic range of the input). Next, the output image sequence consists of both maximum and minimum values corresponding to the illuminant. In this situation, the localization algorithm must be compatible with PSF, a model of negative and positive amplitudes that is likely to produce more accurate localizations. Therefore, the output pixel stream contains more data than the input stream. This is because one output pixel value (or two pixel values if the positive and negative values are separated) is generated for each order of the Haar wavelet function applied to the pixel of a particular input. Because.

For example, if the 1st to 4th order HWK functions are applied to the input pixel stream, the output pixel stream will have 4 (or positive and negative values separated) for each input pixel value. It has 8) pixel values (1 / 2 is generated by each of the 1st to 4th order HWK functions). These four (or eight) values are typically concatenated one after the other in the output pixel stream, and the pixel-by-pixel output pixel stream is used as its input by the localization program 230 for HWK filtering. The illuminant can be localized based on the pixels. In this regard, the localization program 230 receives the pixel-by-pixel output streams 234 from the HWK calculator 228 and emits light to these output streams rather than to the raw pixel streams 232 from the microscope. It is of conventional operation, except that body / phosphor localization is performed. Of course, as another embodiment, a time filter function other than the Haar wavelet function, such as a Butterworth filter or a Chebyshev filter, may be applied. Therefore, the description of the HWK filter herein is merely an example, and more generally, other time filter functions that can distinguish between events that persist for different lengths of time may be used.

FIG. 1 is a flow chart showing in more detail the processing executed in one embodiment of the present invention (however, a further graph example will be described later). Here, as a general schema, each time filter is sequentially applied to each point of a certain pixel luminance time trace before moving on to the next pixel trace. Then, after each output time trace is recombined into the image sequence, this procedure is repeated with the following filter to add the result to the previous result. However, applying each filter in turn for a single point in the pixel time trace, adding each result to the output time trace, and then moving to the next position in the input time trace is also a valid process. .. Otherwise, the results of the filters at the same time point in all pixel luminance traces can be combined to form the result in an image to be added to the output series before moving on to the next point in the time trace. The order in which these integrations are performed is important as long as the entire image series is consistent, or if the order of integration in the output file is known even if it is not consistent. Absent.

From the above viewpoint, FIG. 1 shows an example of a processing flow diagram that can be executed in one embodiment of the present invention, and this processing is a processing that operates for each filter and each pixel position. That is, for the input image sequence, a first time filter, for example, an nth-order HWK filter is applied to each stream of pixels for each pixel, and the processing is performed until all pixel positions are processed by the filter. .. After that, the process is repeated while applying the n + 1th-order filter, but the process is performed again until all the pixel positions are processed for the entire video sequence. After that, this process is repeated for each degree of filter in turn until all the filters are applied, in which case the output image sequence (here it is much longer than the input sequence, specifically). The length of the input sequence is multiplied by x (or 2x), where x is the number of filters of different orders applied) to the localization algorithm for illuminant detection. It is output.

More specifically, with reference to FIG. 1, s. In 1.2, the image sequence of the microscope is s. Received at 1.2, then s. In 1.4, the time filter for processing to be applied (eg, HWK, Butterworth, Chebyshev, etc.) is selected, and the order of the filter to be applied first is selected. Next, s. In 1.6, the position of the first pixel in the received image frame, for example, the pixel position (a, b) in the frame is selected. A time trace or stream of a pixel is then created by examining the pixel values for the positions of the selected pixels in each frame of the input microscopy frame sequence. That is, for example, by concatenating all the pixel values of the positions (a, b) from each frame in the series into one series, one pixel data for the pixel positions (a, b) in the frame. Stream is retrieved.

s. In 1.10, the pixel time trace / stream start point was selected and s. In 1.12., the selected filter is applied to the pixel values in and around the selected starting point. Next, s. At 1.22, the result of the filter (details of the filter calculation will be described later) is acquired, after which the output time trace is started to include the result for a particular pixel position (a, b). s. In 1.24, it is evaluated whether or not the filter is applied to all the pixels (a, b) in the frame series, and if not, s. In 1.26, the next pixel in the series is selected and the process is s. Return to 1.10. By repeating this processing loop, the time filter is applied to all pixels in the time trace / stream of that pixel with respect to the pixel position (a, b).

Some of the filter results may have negative results and should be removed. Then, according to the options described above, a cleanup routine is executed in the pixel time trace, and after processing the negative value by the method described in detail later, as a pixel trace for the position (a, b) in the output series. , Process the output trace of pixel positions (a, b) in the output. Next, the next pixel position in the frame (eg, (a + 1, b) can be processed in the same way, and the process returns to s.1.6. By repeating the above steps for the position of each pixel in the microscopic frame. , A pixel trace for the currently selected time filter is obtained for each pixel position (s. 1.34).

Subsequent processing determines if all the desired time filters have been applied. In this regard, higher order filters, such as first to fourth order Haar wavelet filters, or Butterworth filters, will usually also be applied. If not all filters have been applied, s. At 1.40, the next filter is selected (eg, if the primary HWK filter has just been applied, the secondary filter is selected, etc.) and the process is s. Return to the beginning of the 1.4 loop. In this way, the corresponding pixels at the same position (a, b) in each frame are processed by using the first time filter, and then the corresponding pixels at the same position (a, b) are the first. Processing is performed using a time filter of 2 (eg, a higher order filter) until all filters are applied to each pixel trace. Once all the filters have been applied, a set of filtered pixel traces can be collected, for example by concatenating them in a known order, and then used as input to a localization algorithm.

FIG. 3 shows a demonstration experiment of how filtering of the pixel luminance time trace separates the fluorescence contributions of individual light emitters. The three images above (302, 304, 306) are three in a simulation of two adjacent (ie, separated by smaller intervals than PSF) phosphors that randomly switch between luminescent and non-luminescent states. Shows continuous frames. One fluorophore is located in the center of the pixel marked "a" and the other fluorophore is located in the center of the pixel "c". In the central frame (304), both phosphors are in a luminescent state, in which case the individual PSFs (fluorescent spots) substantially overlap. In the first frame and the second frame (302 and 306, respectively), only the phosphor of "a" emits light. The switching between the two illuminants can be seen in the corresponding pixel luminance time trace (308), which is marked "a", "b", "c" in the images (302, 304, 306). The three pixels are shown as corresponding to frames 79, 80 and 81 of the series. In the frame 76, only the phosphor in the pixel "c" emits light, which means that the brightness of the pixel "c" is higher than the brightness of the pixel "b", and similarly, the brightness of the pixel "b" is the pixel. This is reflected in the fact that it is higher than the brightness of "a". When only the fluorophore in "a" emits light in frames 78, 79, 81 and 82, the opposite ratio (ratio) holds. At frame 80, variations in the pixel luminance time trace are observed. The relative magnitude of this variation between the three pixels at frame 80 is very similar to the variation observed at frame 76 when only the fluorophore at "c" is emitting light. This variation can be separated from the remaining time trace by applying a filter.

The result (310) of using the first Haar wavelet (described in FIG. 4 and below) as a filter function for these three pixels shows only fluctuations that continue only during a single frame. Compared to the case where the phosphor in "a" is switched on (frame 78) and the other cases where the value is close to zero, and even when both are emitting light together in frame 80, "c". When the phosphor in "" is switched to the light emitting state, the brightness ratios of the three pixels are reversed (frames 76 and 80). When all filtered pixel luminance time traces are recombinated into one image sequence, the filtered image (312) corresponding to frame 80 is the size of the PSF centered on pixel "c". Indicates a certain fluorescent spot. By applying a third Haar wavelet as a filter, longer time variability caused by the fluorophore at "a" switching to the luminescent state between frames 78 and 82 is separated. The image (314) from the filtered sequence corresponding to the frame 80 in this case shows a fluorescence "spot" which is the size of the PSF centered on pixel "a". The fluorescence contribution (304) from the two illuminants in frame 80 is separated from each other into two separate image frames (312 and 314). In this state, the standard low-density localization algorithm should be able to localize the positions of the two illuminants / phosphors individually with high accuracy, but in the previous case, the existence of the two illuminants. Is indistinguishable and is therefore mislocalized at a central location between the two phosphors, causing an artifact in the reconstruction. The output from the first Haar wavelet filter contains noise due to illuminant fluctuations that last longer than a single frame of time. In the example (316) of the output frames from this filtered sequence when the fluorophore in "a" emits light for a period of 5 frames (frames 78 to 82), the relative amplitude of this noise determines the PSF. It shows that it is not reflected accurately. However, some localization algorithms often attempt to fit this noise as fluorescent "spots", leading to poor image quality in the reconstruction. The fittings to these localizations are low-strength, not physically wide or narrow, and can be easily removed from the list of localizations used during reconstruction, if desired. The number of overlapping illuminants that can be separated in this way is essentially unlimited, as long as they all result from variations on different time scales, sufficient signal vs. noise is provided.

The filter set can be any set that compares the brightness values of the pixels over a range of adjacent or adjacent frames. Each filter has a characteristic time scale that is sensitive to fluctuations in brightness. The example described from this point of view uses the Haar wavelet function as a filter set. FIG. 4 shows the first four wavelets (402, 404, 406, 408, respectively). These functions have been widely used for image analysis in the data compression phase. When applied to luminance time traces as described herein, these functions show the difference between the mean intensity values before and after a point in the trace on a changing time scale. The pixel time trace marked in FIG. 3B with the first Haar wavelet filter applied is shown for the start frames 80 (410) and 82 (412), as well as the third Haar. The filter applied to the frame 80 (414) is shown.

Here, two examples show the exact form of implementation of the algorithm. Both examples use the above Haar wavelet function as a filter set. The first example shows how to apply a filter to a pixel luminance time trace by matrix multiplication, highlighting the individual steps involved. The following example convolves the time trace with the kernel as a simpler implementation.

ここで、ｘ，ｙは画素インデックス、ｔはフレーム番号である。各画素について、順に、その輝度トレースＸ（ｔ）（列ベクトルとして）にハール行列を乗算して、その変換トレースＹ（ｔ’）を生成する。

ハール行列の非ゼロ要素は以下のように与えられる：

ここで、ｉ，ｊは行と列のインデックスであり、ｆｌｏｏｒ（．．）は最も近い整数に切り捨てることを示し、Ｎはフレームの総数であって２の整数乗になるように切り捨てる必要があるものである。正の整数ｍは、ハールウェーブレット変換（ＨＷＴ）の「レベル」を表す。各レベルは、最短（ｍ＝１）から最長（ｍ＝ｌｏｇ_２Ｎ）までで、元の時間トレースにおいて異なる長さの蛍光体の点滅、及びＤＣ成分に対応する。これは、単一フレームから画像の長さの半分まで続く輝度の変動（点滅）に一定の背景値（試料とカメラ成分の和）を加えたものに対応する。ｍの適切な値については後述する。単純な４フレームの輝度トレースのＨＷＴの例を以下に示す：

ここで、ハール行列の異なるフィルタレベルは、ｍ＝１（下の２行）、ｍ＝２（第２行）、及び背景（最上行）となる。 Matrix multiplication example:
In the first step, a set of pixel luminance traces is created from the unprocessed image sequence.

Here, x and y are pixel indexes, and t is a frame number. For each pixel, the luminance trace X (t) (as a column vector) is multiplied by the Haar matrix in order to generate the transformation trace Y (t').

The non-zero elements of the Haar matrix are given as follows:

Here, i and j are the row and column indexes, floor (...) indicates to truncate to the nearest integer, and N is the total number of frames and needs to be truncate to the integer factorial of 2. It is a thing. The positive integer m represents the "level" of the Haar wavelet transform (HWT). Each level ranges from the shortest (m = 1) to the longest (m = log ₂ N) and corresponds to different lengths of fluorescent flashes and DC components in the original time trace. This corresponds to a change in brightness (blinking) that lasts from a single frame to half the length of the image, plus a constant background value (sum of sample and camera components). An appropriate value of m will be described later. An example of an HWT for a simple 4-frame luminance trace is shown below:

Here, the different filter levels of the Haar matrix are m = 1 (bottom two rows), m = 2 (second row), and background (top row).

フィルタ処理された画素の輝度トレースを取得するためにＨＷＴの逆行列を適用する必要がある。ハール行列の逆行列は、その転置行列（ｔｒａｎｓｐｏｓｅ）と等しいため、簡単に求めることができる。

フィルタ処理された画素の輝度トレースＺ^（ｍ）は次式で与えられる：

下記は、式（４）の変換された輝度トレースに対してフィルタレベルｍ＝１を適用した後の上記の逆変換の例である。

フィルタ処理された輝度トレースには、大きさが等しく符号が反対の奇数／偶数フレームのペアが含まれることに留意すべきである。これは単一ハール周波数成分を使用することによる避けられない結果であるが、符号の空間的相関は物理的に意味がある。ＰＳＦと同じ大きさの正値のクラスタは、奇数フレームから偶数フレームへと発光体の輝度が増加することを示す。反対に、同様の負値の集まりは、発光体輝度が低下することを示す。これは、奇数番号のフレームでは全ての負値をゼロに調整し（ｃｒｏｐｐｉｎｇ）、且つ偶数番号のフレームでは全ての正値をゼロに調整してから−１を乗じることにより、オンとオフの遷移を区別するメカニズムを提供する。フィルタ処理された輝度トレースに対するこの調整手順は、次の条件式で表される：

ここで、Ｚ’^（ｍ）（ｔ）は、調整された画素の輝度トレースである。 In the next step, a filter (level m) is applied to the converted luminance trace. This is done by setting all elements of the column vector that do not correspond to a particular level of the Haar matrix to zero.

It is necessary to apply the inverse matrix of HWT to obtain the luminance trace of the filtered pixels. Since the inverse matrix of the Haar matrix is equal to its transpose, it can be easily obtained.

The brightness trace Z ^(m) of the filtered pixels is given by:

The following is an example of the above inverse transformation after applying the filter level m = 1 to the transformed luminance trace of equation (4).

It should be noted that the filtered luminance trace contains odd / even frame pairs of equal size and opposite sign. This is an unavoidable result of using a single Haar frequency component, but the spatial correlation of the sign is physically meaningful. Positive clusters of the same size as the PSF indicate that the brightness of the illuminant increases from odd-numbered frames to even-numbered frames. On the contrary, a similar collection of negative values indicates that the luminous intensity of the illuminant decreases. This is a transition between on and off by cropping all negative values in odd-numbered frames, and adjusting all positive values to zero in even-numbered frames and then multiplying by -1. Provides a mechanism to distinguish between. This adjustment procedure for filtered luminance traces is expressed by the following conditional expression:

Here, Z' ^(m) (t) is a brightness trace of the adjusted pixel.

This transformation filter inverse transformation process is repeated for all pixels in the image series to generate a set of filtered luminance traces Z _xy ^(m) _{from a set of unfiltered traces X xy.} These are combined again filtered image sequence I _{n ^(m)} is formed.

次に、この新しい画素の輝度トレースの集合Ｘ’（ｔ）に対して、処理が繰り返される。結果である画像系列はその前の画像系列に追加され、フレーム数は元のデータの２倍になる。 For the first filtering level, m = 1, the HWT compares the luminance value with only one of the two adjacent frames. This can be more easily corrected by repeating the process of shifting by one frame (rotating the sequence of the Haar matrix) for all wavelet functions. Similarly, it is also effective to periodically rotate the image sequence by one frame. This will compare even frames with odd frames (2 and 3, 4 and 5 ...). The brightness trace of this new pixel is given as follows:

Next, the process is repeated for the set X'(t) of the luminance traces of the new pixel. The resulting image sequence is added to the previous image sequence, and the number of frames is double that of the original data.

The final output image sequence is obtained by repeating the entire process for each of the desired filter levels m and adding the resulting image sequence together, resulting in a much longer image sequence than the original image sequence. Generated. When the filter level m> 1 is executed, the output of Equation 7 produces 2 ^(m-1) overlapping frames. Therefore, before proceeding to Equation 9, ^{all elements of Z (m)} except the 2 ^(m-1) th can be deleted, resulting in a shorter luminance trace and a shorter image sequence obtained from Equation 11. .. However, since the original image sequence ^{can be permuted 2 (m-1)} times before the output overlap occurs, each filter level still outputs a total of 2N frames.

The appropriate number of filter levels to apply will depend on the blinking characteristics of the phosphor. In essence, one is an attempt to select a time scale range that covers most of the "on" time of the illuminant. Therefore, this procedure can be continued for the 3rd, 4th or higher levels (corresponding to 4,8,16 ... frame comparison), but if the blinking characteristics of the phosphor are poor or buffered. Unless the conditions are far from optimal, the number of illuminants that are "on" long enough after the fifth level is reduced. However, choosing more filter levels has few disadvantages other than increased image sequence size and associated analysis time. In fact, if the total number of original frame N is 10 ⁴ or more, the computational load in the HWT calculations can be very heavy. Therefore, the image sequence can be divided into smaller parts (eg, 256 frames). Each part is processed individually and the result is added to a single image series. This also prevents too many frames from being discarded if N is not a power of 2.

Once the required number of filter levels have been applied, the output image sequence can be analyzed by conventional methods using single or double illuminant fitting algorithms, or by nonlinear image processing techniques such as SOFI or SRRF. The only difference when performing the fitting is that the background has been removed and the camera reference level must be set to zero. In addition, filtering, especially the first (m = 1) filter level, increases the relative contribution of noise (shot noise, readout noise, etc.) to the pixel luminance trace. This can produce relatively bright single pixels in the processed image, especially when the brightness of the illuminant in the original series is low (see 316). Localization algorithms may attempt to fit these as illuminant PSFs in very narrow cases (in the case of a single entity) or in very wide cases (in the case of diffuse clusters). If the algorithm supports it, the image quality of the reconstruction can be significantly reduced by significantly omitting localization using non-physically wide or narrow PSFs as compared to using raw data. Can be improved, but still a larger number of localizations remain than in the case of raw data (Note: as an alternative, a filter is introduced at the output stage of the algorithm to make these locally brighter. You can also delete the pixel). This is because the localization caused by false noise is generally due to lower luminance than the "real" illuminant signal (unless the signal vs. noise is very low). Filtering localization at low brightness is also effective (to a lesser extent) and is compatible with the astigmatism lens method of 3D localized microscopy.

先に述べたように、各画素を個別に輝度時系列として扱う。したがって、画像スタック（ｌ（ｘ，ｙ，ｔ））が与えられると、ある画素位置ｘ，ｙに対して、時系列はＸ（ｔ）＝ｌ（ｘ，ｙ，ｔ）となる。次に、Ｘとカーネルとの畳み込みを計算する：

等、ここで＊は離散的畳み込みの演算子である。Ｘをパディングしないので、結果であるＺはＸよりも短くなる。画素ｘ，ｙに対して、様々なＺを連結することによって最終的な時系列を作成する：

ここで、ｎ_ｉはＺ_ｉにおける要素の数である。蛍光体がオフに切り替わると、Ｚに負値が生じる。最後のステップでは、正値と負値をＺ’に分離する。

代替として、絶対値を使うこともできる：

最終的な時間系列Ｚ’のそれぞれを画像スタックに再構成し、ＴｈｕｎｄｅｒＳＴＯＲＭ等のサードパーティアルゴリズムで各フレームを個別に分析する。 Example of kernel convolution:
The Haar filter can be similarly represented as a one-dimensional image kernel. The sizes of the first three Haar kernels are 2, 4, and 8, respectively, given:

As mentioned earlier, each pixel is treated individually as a luminance time series. Therefore, given an image stack (l (x, y, t)), the time series is X (t) = l (x, y, t) for a certain pixel position x, y. Next, calculate the convolution between X and the kernel:

Etc., where * is a discrete convolution operator. Since X is not padded, the resulting Z is shorter than X. Create a final time series by concatenating various Z's for pixels x, y:

Here, _ni is the number of elements in _{Z i.} When the phosphor is switched off, a negative value occurs in Z. In the final step, the positive and negative values are separated into Z'.

As an alternative, you can use the absolute value:

Each of the final time series Z'is reconstructed into an image stack, and each frame is analyzed individually by a third party algorithm such as ThunderSTORM.

The implementation of this algorithm is very fast and takes much less time than subsequent localization analysis. Combined with the Gaussian fitting of a single illuminant, it not only improves accuracy, but also makes it one of the fastest high-density methods of calculation.

(References of ThunderSTORM) "ThunderSTORM: A comprehensive ImageJ plug-in for PALM and STORM data analysis and super-resolution imaging. , Chrisek, P. et al. , Borkovec, J. Mol. , Svindrych, Z. et al. And Hagen, G. et al. M. Written by Bioinformatics, Vol. 30, pp. 2389-2390, 2014

Various further modifications, either additions, deletions, or substitutions, may be made to the aforementioned implementations to provide further implementations, all of which are included in the accompanying claims. To do.

Claims (15)

A step of receiving a temporal sequence of microscopic image frames captured by a microscope, wherein the image frame is an image containing a luminescent material or a fluorophore.
To obtain each set of temporally filtered pixel values for each of the same pixel positions in the frame, the sequence of the frame is used with a plurality of temporal filters having different filter characteristics. With the step of temporally filtering each of the set of pixel values for each of the same pixel positions in
It comprises a step of providing each of the temporally filtered set of pixel values as an input to a localization algorithm to allow localization of the illuminant or phosphor depending on it. ,
A method for localized microscopy. The time-filtering step
With respect to the pixel position in the image frame, a step of forming a pixel trace corresponding to the pixel value at the same pixel position in the image frame;
With the step of applying the first temporal filter to the pixel value in the pixel trace to obtain the first filtered pixel trace;
A step of applying at least a second temporal filter different from the first temporal filter to the pixel values in the trace to obtain a second filtered pixel trace;
A step of combining the first filtered pixel trace with the second filtered pixel trace into a single filtered output trace for input to the localization algorithm. With;
The method according to claim 1. At least a third temporal filter different from the first temporal filter and the second temporal filter is applied to the pixel values in the trace to obtain a third filtered pixel trace. With steps;
The first filtered pixel trace, the second filtered pixel trace, and the third filtered pixel trace are combined for input to the localization algorithm. Further with a step of making a single filtered output trace;
The method according to claim 2. The first, second, and third filtered pixel traces are concatenated to form a combined output trace.
The method according to claim 2 or 3. The time filter is
i) Haar wavelet kernel;
Selected from groups with ii) Butterworth; or ii) Chebyshev;
The method according to any one of claims 1 to 4. The plurality of temporal filters are temporal filters of the same type but having different temporal characteristics, and the temporal characteristics are continuous around the current frame in which the filtered value of the pixel is currently detected. It is related to the number of frames and contributes to the calculation of the filtered value.
The method according to any one of claims 1 to 5. For each of the sets of time-filtered pixel values to identify the position of the illuminant or phosphor in the input image at a higher resolution than when the microscope was used alone. Further include the step of applying the localization algorithm.
The method according to any one of claims 1 to 6. With a system of microscopy configured to produce computer-readable microscopic image frames;
With a processor;
A computer-readable storage medium that stores computer-readable instructions that cause the processor to perform a predetermined operation when executed by the processor, wherein the predetermined operation is performed.
i) The operation of receiving a temporal sequence of microscopic image frames captured by the microscopic system, wherein the image frame is an image containing a luminescent material or a phosphor.
ii) In order to obtain each set of temporally filtered pixel values for each of the same pixel positions in the frame, a plurality of temporal filters having different filter characteristics are used to obtain the frame. The operation of temporally filtering each of the set of pixel values for each of the same pixel positions in the series;
iii) An operation that enables localization of a illuminant or a phosphor depending on each of the temporally filtered set of pixel values as an input to a localization algorithm; With the computer-readable storage medium;
Localized microscopy system. The operation of filtering in time is
An operation of forming a pixel trace corresponding to a pixel value at the same pixel position in the image frame with respect to the pixel position in the image frame;
The operation of applying the first temporal filter to the pixel value in the pixel trace to obtain the first filtered pixel trace;
An operation of applying at least a second temporal filter different from the first temporal filter to the pixel values in the trace to obtain a second filtered pixel trace;
The operation of combining the first filtered pixel trace and the second filtered pixel trace into a single filtered output trace for input to the localization algorithm. With;
The system according to claim 8. The predetermined operation is subjected to a third filtering process by applying at least a third temporal filter different from the first temporal filter and the second temporal filter to the pixel values in the trace. With the operation of acquiring the pixel trace
The first filtered pixel trace, the second filtered pixel trace, and the third filtered pixel trace are combined for input to the localization algorithm. Further with the behavior of a single filtered output trace;
The system according to claim 9. The first, second and third filtered pixel traces are concatenated to form a combined output trace.
The system according to claim 9 or 10. The time filter is
i) Haar wavelet kernel;
ii) Butterworth; or
iii) Chosen from a group with Chebyshev;
The system according to any one of claims 8 to 11. The plurality of temporal filters are temporal filters of the same type but having different temporal characteristics, and the temporal characteristics are continuous around the current frame in which the filtered value of the pixel is currently detected. It is related to the number of frames and contributes to the calculation of the filtered value.
The system according to any one of claims 8 to 12. The predetermined operation is a time-filtered pixel value to identify the position of the illuminant or phosphor in the input image at a resolution higher than that when the microscope is used alone. Further comprising the operation of applying the localization algorithm to each of the sets.
The system according to any one of claims 8 to 12. A computer-readable instruction that, when executed by a computer, causes the computer to perform the method according to any one of claims 1 to 7.
Computer program.

Images