JP2012078827A - Microscope system, microscopy and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide information about the influence of various adjustable parameters to a user even when the procedure of data acquisition and/or data processing is complex.SOLUTION: This microscope system includes; microscopes (5 to 11) for acquiring data; and a calculation device 2 configured to execute the control of the microscopes during acquiring the data and the data processing of raw data acquired by the microscopes. The microscopes and the calculation device 2 are configured to execute data acquisition and/or data processing based on a value set in each of the plurality of adjustable parameters. The calculation device 2 is configured to selectively output graphic data 16 according to the adjustable parameter selected from among the plurality of adjustable parameters via an optical output device 3. The graphic data 16 indicate the influence of the selected adjustable parameter in at least one step of the procedure as the basis of the data acquisition and/or data processing.

Description

  The present invention relates to a microscope system, a microscope method, and a storage medium storing computer-executable instruction codes for the microscope system. In particular, the present invention relates to a system and method for acquiring raw data and computationally processing the acquired raw data with a procedure including a plurality of steps.

  Microscopes have been used in many applications for a long time to observe organic and non-organic materials. Throughout most of that period, the resolution achievable with an optical microscope was limited by the diffraction limit.

  More recently, techniques have been developed that enable the realization of high-resolution microscopes that provide resolutions that are smaller or much smaller than the diffraction limit. Examples of such techniques include techniques that employ structured illumination of the object or luminescence microscopy, and these techniques can identify the position of the object with high accuracy. Non-Patent Document 1 presents an overview of such technology. As a further illustration, such a technique is also described in US Pat. By using such a technique, it has become possible to specify the position of the object with an accuracy in the range of several nanometers to several tens of nanometers. A method that makes it possible to use a high-resolution luminescence microscope for three-dimensional imaging is described in "Method and microscope for performing three-dimensional resolution-enhanced microscopy". ) Patent Document 3, entitled “High-resolution microscope and image splitter arrangement”, and Patent Document 4, entitled “High-resolution microscope and method of determining object positions in two or three” Patent Document 5 entitled “Dimensions (High Resolution Microscope and Method for Identifying Object Position in 2D or 3D)”, all of which have been assigned to the assignee of the present application.

  A feature of various microscopy techniques that allows achieving resolutions much smaller than the diffraction limit is that data acquisition and data processing can be complex. In data acquisition, a plurality of individual two-dimensional (2D) frames are acquired from the object, which are processed to calculate the resulting image data of the object. As an illustrative example, a method employing structured illumination of an object uses one or more 2D frames for each of these different orientations using illumination patterns in different orientations. Can do. By processing information included in these various two-dimensional frames with a computer, an image of the object is generated.

  As a further illustrative example, high resolution luminescence microscopy can use photoswitchable or photoactivatable molecules. For each acquired frame, only a small number of molecules are detectable for each individual 2D frame acquired to ensure that the detectable molecules are not too far away from each other. Can be switched to. A large number of frames, such as 10,000 to 20,000 frames, may be required to combine frames into a substantially complete representation of the observation object.

  Such high resolution methods allow the user to define adjustable parameter values in order to further improve the resolution of the resulting image by computationally processing the raw data. There may be. For methods that employ structured illumination, examples of such adjustable parameters include illumination patterns (eg, pattern periodicity), or the number of different orientations of the illumination pattern from which a frame is acquired. In the case of a method using a luminescence microscope, examples of such tunable parameters include the fit mask used to computationally evaluate the frame to locate the luminescent event, or the spectrum of the switching signal and Strength is included. In the case of a method in which frames are acquired at several different object planes, the adjustable parameters can include the positions of those object planes measured along the axial direction of the microscope, for example. Further examples of adjustable parameters include the characteristics of filters used to filter images in position space or Fourier space.

  Due to the complexity of the procedures underlying data acquisition and processing, it can be difficult for the user to understand the importance and impact of the various tunable parameters on the underlying procedure. Understanding such effects is important for the user, especially when the user sets the value of an adjustable parameter for conducting experiments or processing raw data.

  The way in which the effects of adjustable parameters on the resulting image are shown may help to some extent the user understand the importance of the various adjustable parameters. Such simulation methods may use previously acquired data or typical data stored in memory. Patent Document 6 describes an example of such a method. Performing data processing using the currently set values for the various tunable parameters may help the user identify settings that are appropriate for the structure currently being observed. However, it does not convey information about the underlying procedure. Such information can help users understand the importance and impact of various tunable parameters and use such understanding when performing data acquisition and data processing on new objects. Therefore, it becomes particularly effective. In addition, when a simulation of data processing of acquired data is executed using the method described in Patent Document 6, raw data must first be acquired. The results of such a simulation are of limited value when planning data acquisition. In particular, if data acquisition can be performed only once for a given sample, such as when irreversible molecular light switching is performed in data acquisition with a luminescence microscope, the user can A better understanding of the importance and impact of the various adjustable parameters is also desirable.

European Patent No. 1157297 German Patent No. 102006021317 German Patent Application Publication No. 102009043744 German Patent Application Publication No. 102009060490 German Patent Application Publication No. 102009060793 German Patent Application Publication No. 102007046469

L. Schermelleh et al. "A Guide to super-resolution fluorescence microscopy" J. Cell Biol. 190 (2): 165 (2010)

  An object of the present invention is to provide a microscope system, a microscope method, and a storage medium for storing computer-executable instruction codes, which assist a user in planning data acquisition and / or data processing. In particular, it is an object of the present invention to provide a system or method that provides a user with information about the effects of various adjustable parameters, even when the data acquisition and / or data processing procedures are complex.

  According to the present invention there is provided a storage medium storing a microscope system, a microscopy method and computer-executable instruction code as defined in the independent claims. The dependent claims define embodiments.

  According to one aspect, a microscope system is provided. The microscope system includes a microscope for data acquisition and a calculation device. The microscope has at least one controllable component. The computing device is configured to perform control of the at least one controllable component of the microscope during data acquisition and / or processing of raw data acquired and provided by the microscope. The microscope system may be configured to perform a procedure including multiple steps to perform data acquisition and / or data processing. The microscope and the calculation device are configured to execute data acquisition and / or data processing according to values set for the plurality of adjustable parameters, respectively. The microscope system includes a light output device. The light output device is connected to the computing device. The computing device is configured to selectively output graphic data to the light output device in response to the adjustable parameters, the adjustable parameters being selected in a user-defined manner. The graphic data output via the light output device represents the influence of the selected adjustable parameter in at least one step of the procedure underlying data acquisition and / or data processing.

  Such a microscope system is configured to provide a user of the microscope system with information describing the operation of the procedure. Since the graphical data is selectively output in response to the selected adjustable parameter, the user can selectively retrieve information about the effects and effects of the various adjustable parameters. This makes it easy for the user to control a microscope system in which a plurality of adjustable parameters can be set.

  The computing device may be configured to calculate image data from a plurality of frames collected in data acquisition. In this case, the graphic data may be output via the light output device for at least one selectable adjustable parameter, the graphic data output at this time being in a procedure executed for data processing. , Representing the effect of this adjustable parameter. This helps the user to understand the underlying algorithms for data processing.

  The microscope system may include an input device that allows a user to set the values of various adjustable parameters. The computing device may be configured to automatically output graphic data for one of the adjustable parameters currently being set by the user via the input device. This allows the user to receive information about that one of the tunable parameters that are currently being adjusted for the relevant value.

  The computing device may be configured to adjust the graphic data in response to user input. This allows the user to retrieve additional information about the selected adjustable parameter in response to user input in an interactive manner.

  The computing device may be configured to output graphic data in response to a value set for the selected adjustable parameter. The graphic data may be adaptively adjusted when the set value of the selected adjustable parameter is adjusted by the user. This assists the user in identifying values for the selected adjustable parameters that are appropriate for later operating the microscope system for data acquisition or data processing.

  The computing device may be configured to output graphic data in response to a value set in yet another adjustable parameter that is different from the selected adjustable parameter. This allows taking into account the interrelationship between different adjustable parameters. As an illustrative example, if the graphic data represents a Fourier space that is accessible using a structured illumination pattern, the set value for the periodicity of the structured illumination pattern and different orientations of the illumination pattern Graphic data may be generated and output in accordance with another set value for the number and both. If either the periodicity or the number of different orientations is changed by the user, the graphic data may be adjusted accordingly.

  The computing device may be configured to output graphic data for at least one of the selectable adjustable parameters before data acquisition or data processing is performed. This allows the user to utilize the graphic data prior to subsequent data acquisition or data processing.

  Graphic data may be generated to convey various different types of information to the user. In one implementation, the computing device generates graphical data that represents the effect of the selected adjustable parameter on the speed and / or resolution and / or sensitivity achievable in performing data acquisition and / or data processing. It may be constituted as follows. Such information can be used when designing experiments. Alternatively or additionally, the graphic data includes information about the data processing procedure or which step of the data acquisition or data processing procedure is affected by the selected adjustable parameter. May be. The graphic data may include graphic information that displays the effect of a filter or fit mask in the data processing. The graphic data may include information that displays the effect of the illumination pattern for structured illumination. The graphic data may include information indicating the effect of exposure time or power density of the light switch signal for methods using light switchable or photoactivatable molecules. There are various examples of such procedures, including a technique called “Photo Activated Localization Microscopy” (PALM). The graphic data may include information about the separation of the data acquisition channel.

  When graphic data is generated and output, additional graphic information can be output in addition to the graphic data associated with the selected adjustable parameter, even if raw data has already been acquired. . The further graphic information may be generated in response to the raw data collected in the data acquisition. In another implementation, the acquired raw data can be combined with the graphic data or incorporated into the graphic data.

  The microscope system may be configured to acquire data using structured illumination. The microscope system may include an apparatus for generating an illumination pattern. The apparatus for generating the illumination pattern may be configured so that the orientation of the illumination pattern can be changed. At least one of the adjustable parameters that can be selected to output the associated graphic data is the periodicity of the illumination pattern, the number of different orientations of the illumination pattern, and the filtering of the acquired raw data May be included in a group consisting of filter functions used in

  The microscope system may be configured to perform microscopy using photoactivatable or photoswitchable molecules. As an illustrative example, the microscope system may be configured to perform a PALM method. The microscope system may include a switching signal source. The microscope system may include an optical switching signal source. At least one of the adjustable parameters that can be selected to output the associated graphic data is the size of the mask used in the fit to locate the molecule, the spectrum of the switching signal, and The intensity of the switching signal may be included in the group. As an illustrative example, the graphic data may display the size of the fit mask in relation to the acquired raw data. The graphic data may additionally or alternatively display how the spectrum or intensity of the switching signal affects the transition rate between different states of the colorant or pigment. . The graphic data may include a Yabronsky diagram to display this information.

  The microscope system may be configured to perform microscopy using photoactivatable or photoswitchable molecules. The microscope system may include a switching signal source. The microscope system may be further configured to perform data acquisition for a plurality of object planes that are spaced apart from each other. The microscope can include components for adjusting the position of the object plane. The component for adjusting the position of the object plane may include an adjustable lens or an adjustable microlens array. Alternatively or additionally, the component for adjusting the position of the object plane may comprise an image splitter device. The image splitter device can guide various components of the detected light to different optical paths with adjustable optical path lengths, where the different optical paths are controlled to adjust the position of the object plane. . The computing device may be configured to generate graphic data according to the position set for the object plane. Graphic data may be generated to show how the position of the object plane affects the size of the measurement area and / or the achievable resolution in the axial direction of the beam path .

  According to another aspect, microscopy is provided. In this method, data acquisition is performed, and thereby raw data is acquired. Subsequently, data processing is performed, whereby the obtained raw data is processed computationally. Data acquisition and / or data processing is performed according to values set in the plurality of adjustable parameters. Data acquisition and data processing may be performed according to a procedure that includes multiple steps. Graphic data is selectively output to the light output device. The graphic data is selected in response to an adjustable parameter selected from the plurality of adjustable parameters in a user-defined manner. The graphic data is associated with the selected adjustable parameter. The graphic data represents the influence of the selected adjustable parameters in at least one step of the above procedure that is the basis for data acquisition and / or data processing.

  As described in connection with the microscope system, this microscopy may use a variety of implementations.

  This microscopy may be performed using a microscope system according to any one of the aspects or embodiments described herein.

  According to another aspect of the present invention, a non-transitory storage medium storing a computer program is provided. A computer program includes a sequence of computer-executable instructions. When these series of instructions are executed by the computing device of the microscope system, these instructions direct the microscope system to perform the method of any one of the aspects or embodiments described herein.

  The systems and methods according to embodiments may be used for complex microscopy. In particular, the systems and methods according to embodiments may be used for microscopy where multiple frames are first acquired and subsequently processed to determine image data computationally.

  The various features and effects achieved by the embodiments of the present invention, and the manner in which these effects are achieved, will be further described with reference to exemplary embodiments described in detail with reference to the drawings. Describe in detail.

FIG. 1 is a schematic diagram of a microscope system according to an embodiment. FIG. 2 is a schematic diagram of a microscope system according to another embodiment. FIG. 3 schematically illustrates a graphical user interface of a microscope system according to an embodiment. FIG. 4 schematically shows a graphical user interface of the microscope system according to the embodiment shown in FIG. FIG. 5 schematically shows a graphical user interface of the microscope system according to the embodiment shown in FIGS. 3 and 4. FIG. 6 schematically shows a graphical user interface of a microscope system according to another embodiment. FIG. 7 schematically shows a graphical user interface of a microscope system according to another embodiment. FIG. 8 schematically shows a graphical user interface of a microscope system according to another embodiment. FIG. 9 schematically shows a graphical user interface of the microscope system according to the embodiment shown in FIG. FIG. 10 shows the operation of the microscope system for three-dimensional emission imaging. FIG. 11 schematically shows a graphical user interface of a microscope system according to another embodiment. FIG. 12 schematically shows a graphical user interface of a microscope system according to another embodiment. FIG. 13 schematically shows a graphical user interface of the microscope system according to the embodiment shown in FIG. FIG. 14 schematically shows a graphical user interface of a microscope system according to another embodiment. FIG. 15 schematically shows a graphical user interface of the microscope system according to the embodiment shown in FIG. FIG. 16 schematically shows a graphical user interface of a microscope system for performing luminescence microscopy according to another embodiment. FIG. 17 schematically shows a graphical user interface of the microscope system for performing the luminescence microscopy according to the embodiment shown in FIG. FIG. 18 schematically shows a graphical user interface of a microscope system for performing luminescence microscopy according to another embodiment. FIG. 19 schematically illustrates a graphical user interface of a microscope system for performing luminescence microscopy according to another embodiment.

  The embodiment of the present invention will be described in more detail. The description of the embodiments should in no way limit the present invention, but should be construed as merely illustrative. Some embodiments are described in the context of specific microscopy techniques, such as in the context of techniques that employ structured illumination ("Structured Illumination Microscopy", SIM) and / or PALM methods. However, the embodiments are not limited to these specific application areas. The features of the various embodiments can be combined with one another unless explicitly excluded from the following description.

  FIG. 1 is a schematic diagram of a microscope system 1 according to an embodiment. The microscope system 1 has a calculation device 2. The computing device 2 controls at least one controllable component of the microscope, and processes the frames acquired by the microscope, thereby computationally combining a plurality of frames acquired by the microscope to obtain an image of the object. . The computing device 2 is connected to the light output device 3. The light output device 3 may be a display. The computing device 2 is configured to output graphic data 16 to the light output device 3, as will be described in more detail, which provides information about the procedures underlying data acquisition and / or data processing. Provide to users.

  The microscope system 1 is configured to generate an image of a sample with improved resolution. The microscope system 1 may be configured to generate an image of the photoactivatable molecule by detecting the emission signal. Examples of such procedures and techniques are known in the art under the following names: PALM ("Photo Activated Localization Microscopy"), FPAL ("Fluorescence PhotoActivation Localization Microscopy") Photoactivated localization microscopy))), STORM ("Stochastic Optical Reconstruction Microscopy"), SPDM ("Spectral Precision Distance Measurement"), PALMIRA ("PALM with Independently Running Acquisition "), GSDIM (" Ground State Depletion and Individual Molecular return "), or dSTORM (" direct STORM "). These are exemplified in Non-Patent Document 1 as a further reference.

  A plurality of frames of the sample 19 are acquired by the microscope. Sample 19 can be marked with the dye DRONPA (see WO 2007/009812) or the dye DENDRA (see Gurskaya et al., Nature Biotech. Vol. 24, pp. 461-465, 2006). For the activation and luminescence excitation of the dye molecules, the microscope has a radiation source 5 comprising at least one laser 12,13. In an exemplary embodiment where the wavelengths of activating radiation and luminescence activation may be different, the radiation source 5 comprises two lasers 12,13. The output beams of the two lasers 12 and 13 may be combined by a beam combiner 14. One of the lasers 12 and 13 may output radiation (activation beam) with a wavelength of 405 nm. The other of the lasers 12 and 13 may output 488 nm wavelength radiation (luminescence excitation and deactivation). If dyes are used that can be activated and luminescent excited at the same wavelength, the radiation source 5 may comprise only one laser. The dye DENDRA is a typical example of a dye that can perform molecular activation and luminescence excitation at the same wavelength.

  An optical filter 15 is provided downstream of the beam combiner 14. The optical filter 15 may be an acousto-optic filter 15. The optical filter 15 is used for wavelength selection. The optical filter 15 is also used to quickly switch the laser wavelength output from the radiation source 5 and / or to selectively attenuate one of the plurality of laser wavelengths output from the radiation source 5. It is done. The beam output from the radiation source 5 is focused by the lens 6 or other optical components, and enters the pupil of the objective lens 8 via the dichroic beam splitter 7. The lens 6, the dichroic beam splitter 7, and the objective lens 8 are configured to cause the radiation beam output from the radiation source 5 to enter the sample 19 as wide-field illumination.

  The luminescent radiation output from the sample 19 is collected on the objective lens 8. The dichroic beam splitter 7 is configured to pass luminescence radiation. The luminescent radiation impinges on the detector 11 via the filter 9 and the tube lens 10. The detector 11 may be a matrix detector. The detector 11 may be a CCD detector.

  The calculation device 2 is connected to a microscope. In the microscope system 1, the calculation device 2 performs both control of the operation of the microscope and processing of raw data acquired by the microscope. In order to control the microscope, the computing device 2 may be connected to a radiation source 5, for example. The computing device 2 may be configured to control the intensity or spectrum of radiation output by the radiation source 5 towards the lens 6. This allows one of the radiation used for dye activation or the radiation used for luminescence excitation to be selectively output. The microscope may include additional components that are also controllable. As an example for explanation, a device for adjusting the observation depth may be provided. Thereby, data acquisition by PALM can be selectively performed on different planes of the object. In one implementation, the apparatus for selecting the observation depth may perform spatiotemporal pulse shaping. This allows depth resolution to be achieved with multiphoton excitation as described in D. Oron et al., “Scanningless depth-resolved microscopy”, Optics Express 13, 1468 (2005). Can be achieved. Other configurations and implementations of the controllable components of the microscope can be used in yet another embodiment.

  The calculation device 2 is further configured to calculate image data of the object 19 from a number of frames. As an illustrative example, more than 10,000 frames may be computationally processed to generate image data of the object 19. The computing device 2 may perform a nonlinear mathematical operation on the acquired emission signal. For this purpose, the calculation device 2 may calculate a power function or exponential function of the acquired signal intensity for each of a plurality of pixels of the acquired frame. Alternatively or additionally, the computing device 2 can collate the Gaussian fit function with the measured intensity distribution, thereby identifying the position of the molecule within the acquired frame. If the microscope is further configured to perform three-dimensional (3D) luminescence microscopy, the computing device 2 may further automatically group the molecular images. Thereby, light emission signals corresponding to each other are identified in different frames associated with different object planes.

  Therefore, the final image of the object 19 obtained as a result in the microscope system 1 is generated by a complicated procedure, and the details of the data processing that is the basis thereof may not be known to the user. is there. The procedure relies on a number of adjustable parameters, and the user can set various values for them. These tunable parameters are filter signal strengths and spectra for dye molecules in the sample, the size of the fit mask used, or a filter function that filters the frame to suppress image noise or the like. May be included. The user may set the values of these various adjustable parameters with the input device 4. As will be described in more detail below, the computing device 2 outputs information to the light output device 3 in order to display the effects of adjustable parameters in the procedure underlying data acquisition and / or data processing. The data may be output automatically or in response to a dedicated user operation.

  FIG. 2 is a schematic diagram of a microscope system 21 according to another embodiment. Among the constituent elements of the microscope system 21, those corresponding to the constituent elements of the microscope system 1 are shown with the same reference numerals as those in FIG. The microscope system 21 is configured to image the sample with improved resolution using structured illumination.

  The microscope system 21 includes an illumination device 22, at least one pattern generator 23, an illumination lens 24 or other optical components, a sample holder 25, an imaging optical system 26, and a detector 11. The components 22, 24-26 and 11 may have a configuration as known from light microscopy and optical measurement and analysis techniques. As an illustrative example, components 22, 24-26 and 11 may be configured as used in an optical spectrometer.

  The pattern generator 23 is generally configured to generate an illumination condition having a predetermined spatial pattern when colliding with the object 19. The pattern generator 23 may be or include a mask having predetermined one-dimensional or two-dimensional transmission characteristics. The mask can be formed by a diffraction grating or a phase grating. The mask may include a plurality of reflective or transmissive pixels in a matrix array. The pattern generator may include a “digital mirror device” (DMD) or liquid crystal matrix configuration. Alternatively or additionally, the pattern generator 23 may include a mirror configuration for generating an interference pattern. Although the pattern generator 23 is shown as being disposed between the illumination light source 22 and the lens 24 in FIG. 2, a pattern generator may be disposed between the lens 24 and the object 19. The pattern generator 23 may be provided directly on the object 19.

  In order to control generation of various different patterns, the calculation device 2 is connected to the pattern generator 23. The calculation device 2 is further configured to reconstruct an image of the object 19 from a plurality of frames acquired by the detector 11 for different positions and different orientations of the illumination pattern. The calculation apparatus 2 may implement the method described in Patent Document 1 in order to reconstruct an image of the object. The computing device 2 may implement other suitable methods to computationally combine multiple frames acquired for different positions or orientations of the illumination pattern to obtain one result image of the object. .

  In the microscope system 21, a final image of the resulting object 19 is generated using a complex procedure. The procedure relies on various adjustable parameters, and their values may be set in a user-defined manner. The adjustable parameters may include the periodicity of the illumination pattern, the number of different orientations of the illumination pattern, or a filter function that filters the acquired frames. The user may set the values of these adjustable parameters with the input device 4. As will be described in more detail below, the computing device 2 is configured to output graphic information to the light output device 3. The output graphical data displays the effect of adjustable parameters on the procedure underlying data acquisition and / or data processing. The computing device 2 may be configured to automatically output graphic data or to output in response to a dedicated user operation.

  According to various embodiments, the computing device 2 of the microscope system outputs information to the user. This information may in particular be graphic data. The graphical data may indicate how any one of various user settings of the plurality of adjustable parameters affects the data acquisition and / or data processing procedures. This helps the user to intuitively understand the operation of the microscope system. The user may be provided with information about the procedure when the user is in the process of setting various adjustable parameter values to perform data acquisition or data processing of the acquired raw data. it can. The processing of the acquired raw data may include processing of a plurality of frames using a computer, thereby generating image data of the object. Such processing is employed in microscopy using structured illumination, or in the PALM method and variations thereof.

  The information output to the user by the microscope system to display the effect of the adjustable parameters can in particular include graphic data. The graphic data may be animation data. The graphic data may be included in the video sequence or may be associated with the video sequence. The graphic data may be selectively output depending on which one of the plurality of adjustable parameters has been selected by the user to set a value assigned to the adjustable parameter. Also, the graphical data can be selectively transmitted in response to a dedicated user request for information about a particular one of the tunable parameters without requiring the user to adjust the value assigned to that tunable parameter. It may be output. Graphic data may be output interactively, in which case the graphic data is adjusted in response to user input. Graphic data may be generated to display the effect of the selected adjustable parameter as the user adjusts the value of the adjustable parameter. In addition to the graphic data, the output information may also include audio data.

  In some implementations, the graphic data for the at least one adjustable parameter used to describe data acquisition and / or data processing is independent of the acquired raw data. This allows information to be provided before any raw data is acquired.

  In some implementations, the preview image can be calculated computationally from the raw data and output in addition to the graphic data. The preview image may indicate how the current set values of the various adjustable parameters affect the image data resulting from the data processing.

  By outputting such graphic data, information is provided to the user to assist the user in the process of planning data acquisition and / or data processing. As an illustrative example, the graphic data output to assist the user in the process of planning data acquisition is a graphic that shows the effect of adjustable parameters on the speed, sensitivity, and resolution of the data acquisition procedure. It can be data. Alternatively or additionally, graphic data describing the data processing procedure may be provided. This helps the user to select values for various parameters such as exposure time, laser power, parameters for image acquisition using structured illumination (ie, SIM method), PALM method parameters, and the like. In order to assist a user planning data processing, graphic data may be output that displays the operation of a filter used for SIM or PALM data processing. Alternatively or additionally, graphic data may be output that displays the operation and effect of the fit mask used to identify the position of the molecule by the PALM method.

  Hereinafter, a method for realizing a graphical user interface will be described with reference to FIGS. 3 to 9 and 11 to 19. Such a user interface can be used in the microscope system 1 of FIG. 1 or the microscope system 21 of FIG. Depending on which one of the plurality of adjustable parameters the user has selected and requested information about, selectively select one of the plurality of graphic representations to explain the underlying technology Can be output. The microscope system may be configured to sequentially output different graphic data when the user selects different adjustable parameters one after another and requests information about them. The output of any graphic data can be realized as described below with reference to FIGS. 3 to 9 and FIGS.

  3-5 schematically show a graphical user interface. The graphical user interface may be output to the light output device 3 of the microscope system. The microscope system may be configured to acquire data by the SIM method. 3-5 show the output of graphic data representing the volume of k-space. Each displayed volume of k-space can be a volume of inverse (ie, k) space that can be sampled using structured illumination.

  The graphical user interface has a display area 31 for displaying the value currently set for the first adjustable parameter. The graphical user interface displays an adjustment element 32 that allows the user to adjust the value of the first adjustable parameter. The graphical user interface includes a display area 33 for displaying the setting value of the second adjustable parameter, and an adjustment element 34 that allows the user to adjust the setting value of the second adjustable parameter. Have. As an illustrative example, the first adjustable parameter may be the periodicity of the illumination pattern. The first adjustable parameter can be the line spacing of the illumination pattern. The second adjustable parameter may be the number of different orientations of the illumination pattern from which raw data is acquired.

  Graphic data 35 is output to the graphical user interface. The output graphic data 35 shows how the first tunable parameter and the second tunable parameter affect the data acquisition and data processing procedure by the SIM. As shown in FIGS. 3-5, the graphic data 35 represents the volume of k-space that is sampled when the SIM procedure is performed with the periodicity of the illumination pattern and the number of different orientations according to the settings by the user. Represents. The calculation device of the microscope system may calculate the boundary of the k-space volume according to the periodicity and the number of orientations of the illumination pattern set by the user. The calculation device can control the light output device 3 according to the calculated boundary. The calculation device may further control the light output device 3 so that additional information is output. As shown in FIGS. 3 to 5, a graphic display 37 corresponding to the isotropic resolution can be output. This assists the user in determining the values of the first tunable parameter and the second tunable parameter so that the resulting resolution is approximately isotropic. Alternatively or additionally, a graphical display 37 representing the desired target resolution may be generated. The graphic representation 37 in this case shows at least the volume of k-space that must be sampled in order to be able to decompose the structure at the target resolution.

  The graphic data 35 may be generated according to values set for the first adjustable parameter and the second adjustable parameter, respectively. As one of these values is adjusted, the graphic data 35 may be adjusted accordingly.

  FIG. 4 shows a typical example of how the graphic data is changed when the setting value of the spatial periodicity of the illumination pattern is changed by the user. The boundary of the k-space volume covered by the SIM procedure that is output to the user shifts from line 36 in FIG. 3 to line 38 in FIG. FIG. 5 shows a typical example of how the graphic data is changed when the user adjusts the value of the number of different orientations of the illumination pattern. The boundary of the k-space volume output to the user shifts to line 40.

  Graphic data 35 may be automatically output when the user activates one of the adjustment elements 32, 34. In other words, when the user adjusts one of the first adjustable parameter (eg, periodicity) or the second adjustable parameter (eg, the number of different orientations of the illumination pattern), the graphic data 35 may be automatically output. Alternatively or additionally, graphical data 35 is selectively output in response to a user operation indicating that the user is requesting information about a procedure for performing data acquisition or data processing. You may do it.

  FIG. 6 schematically shows a graphical user interface output to the light output device 3 of the microscope system. The graphical user interface includes graphic data that displays the effect of the filter in data processing. The microscope system may be configured to perform microscopy according to a SIM procedure, PALM procedure, or other procedure in which the acquired raw data is filtered.

  The graphical user interface has a display area 41 for displaying the value of the adjustable parameter and an adjustment element 42 for changing the set value of the adjustable parameter. As an illustrative example, the tunable parameter may be a cutoff frequency or line width of a filter used for raw data filtering. Various microscopic methods use raw data filtering to suppress noise. The raw data may be frames acquired in a SIM procedure or PALM procedure. Providing information about the filter or filter characteristics can assist the user in finding an appropriate setting for the tunable parameter. This can reduce the risk of the user selecting an adjustable parameter value that leads to information loss.

  The graphical user interface includes graphical data 43 that displays the effect of adjustable parameters in the filtering step of the data processing procedure. In some implementations, the filter characteristics or the transfer function of the filter may be displayed as a function of spatial frequency. When the value of the adjustable parameter is changed according to the user input, the graphic data 43 may be adjusted to adapt to the new set value.

  The filter characteristics may be set by the user after raw data has already been acquired. In this case, the preview image 44 may be generated using the raw data acquired previously. The preview image 44 may be generated using a value set in the filter characteristic. The preview image 44 may be output to a graphical user interface in combination with the graphic data 43.

  FIG. 7 schematically shows a graphical user interface that can be output to the light output device 3 of the microscope system. The graphical data output by this graphical user interface shows and explains how the data processing procedure is executed. The microscope system where the output of this graphical user interface takes place performs microscopy according to a SIM procedure, a PALM procedure, or other procedure where complex data processing is performed by a data processing procedure that includes a plurality of different processing steps. It may be constituted as follows. The graphical user interface has a display area 41 for outputting the value of the adjustable parameter and an adjustment element 42 that allows the user to adjust the set value of the adjustable parameter.

  Graphic data 45 is output to the graphical user interface. Graphic data 45 schematically illustrates a data processing procedure. Any one of a variety of different representations of the data processing workflow can be used. As an example for explanation, a flowchart expression as shown in FIG. 7 may be used. Alternatively or additionally, a graphical representation can be used that displays the effects of the various steps of the workflow in Fourier space or image space.

  The graphic data 45 may be generated and output according to adjustable parameters selected by the user. As an illustrative example, graphic data 45 is generated and output for a procedure that includes a plurality of steps 46-56, highlighting steps 49, 51, 53 that are directly influenced by the selected adjustable parameters. You may do it. Text information may be incorporated into the graphic data 45 to help understand the graphic data 45. As an illustrative example, block 46 may include text information “SIM data processing”. Blocks 47 to 51 are steps in which each processing step is a pre-processing (block 47), which is to remove background signals (block 48), perform scaling (block 49), and separate different imaging orders for SIM imaging. (Block 50) and text information explaining the step of storing the intermediate result (Block 51) can be included. Blocks 52-56 may include text information explaining that each block corresponds to a main process (block 52), in which different orders of k-space images are shifted (block 53). ), Mixing information is determined according to the signal-to-noise ratio (block 54), and filtering (block 55) and inverse transformation (block 56) are performed. Graphic data 45 may include additional blocks for post processing. Post-processing may include clipping, isotropic decomposition, and color channel alignment steps.

  8 and 9 schematically show a graphical user interface output to the light output device 3 of the microscope system. The displayed graphic data is generated and output to explain the effect of the fit mask in determining the position of the molecule. A microscope system using this graphical user interface may be configured to perform microscopy according to a PALM procedure or other procedure in which a fit mask and acquired raw data are matched. Examples of such procedures include luminescence microscopy procedures where a fit mask is superimposed on the raw data to locate the molecules.

  The graphical user interface has a display area 61 for displaying the value of the adjustable parameter. The graphical user interface has an adjustment element 62 that allows the user to adjust the settings of the adjustable parameters. The adjustable parameter can be the size of a Gaussian fit mask. Such a fit mask can be used in the data processing of the PALM procedure or in an associated microscopy procedure. By outputting information about the fit mask via the graphical user interface, it is possible to assist the user in selecting an appropriate fit mask.

  Graphic data 63 output via the graphical user interface schematically illustrates the effect of the size of the fit mask used to locate the molecule. When the value of the adjustable parameter is changed, the graphic data 63 may be adaptively changed to correspond to the new value of the fit mask size set by the user. In the graphic data 63, the fit mask can be superimposed on the camera signal or a portion of the camera signal, thereby assisting the user in selecting an appropriate size of the fit mask.

  The graphic data 63 includes a camera signal 64 or 66 of one molecule. When the graphic data 63 is output after the raw data is acquired, the camera signal 64 or 66 may be data collected in the raw data acquisition. Alternatively, the computing device may comprise a library of typical data that is incorporated into the graphic data 63. Typical data may represent a characteristic image of the molecule depending on the values set for various adjustment parameters that affect data acquisition. The computing device may also be configured to generate typical data 64 or 66 using a simulation process. In FIGS. 8 and 9, the difference in luminance level is shown using hatching lines having different densities.

  The graphic data 63 further includes a fit mask display 65. This fit mask display 65 is shown superimposed on the camera signal. When the user adjusts the value of the adjustable parameter using the adjustment element 62, the fit mask display 65 is changed accordingly.

  With reference to FIGS. 10 and 11, the output of graphic data will be described in the context of a luminescence microscope method and system that enables the generation of three-dimensional images. The three-dimensional PALM procedure is a typical example of such a method and a corresponding microscope system. The microscope system may include a device for selecting the observation depth in order to enable generation of such a three-dimensional image.

  FIG. 10 shows a method and microscope that allow the implementation of three-dimensional luminescence microscopy. In order to perform multiphoton excitation of the dye molecule, the sample may be irradiated with light through the objective lens 71 of the microscope. A device for spatiotemporal pulse shaping may be provided to adjust the observation depth. Thereby, it is possible to acquire one or a plurality of two-dimensional images for each of the plurality of object planes 72 and 73. Inset 74 of FIG. 10 schematically shows a case where there are three fluorescent molecules. The two-dimensional image 75 acquired for the object plane 72 shows an image in which the three molecules have different sizes and intensities. This difference in size and intensity is caused by the different positions of the molecules along the axial direction of the microscope. The two-dimensional image 76 acquired for the object plane 73 also shows images of molecules having different sizes and intensities. This difference in size and intensity is also caused by the difference in molecular position along the axial direction of the microscope. This information may not only be used to identify the lateral position of the molecule, but also allows the position of the molecule to be identified along the axial direction of the beam. There is a correlation between the achievable axial resolution and the axial length that can be measured.

  FIG. 11 schematically shows a graphical user interface that can be output to the light output device 3 of the microscope system. The graphical user interface includes graphical data that shows the effect of the position of the object plane on the axial resolution and the length of the axial measurement area. The microscope system may be configured to perform microscopy according to the PALM procedure for 3D imaging.

  The graphical user interface has a display area 81 for displaying the value of the first adjustable parameter. The graphical user interface further displays an adjustment element 82 that allows the user to adjust the value of the first adjustable parameter. The graphical user interface has a display area 83 for displaying the second adjustable parameter value. The graphical user interface has an adjustment element 84 that allows the user to adjust the setting of the second adjustable parameter. As an illustrative example, the first adjustable parameter may represent the position of the first object plane in 3D microscopy along the axial direction of the microscope. The second adjustable parameter may represent the position of the second object plane in the three-dimensional microscopy along the axial direction of the microscope.

  Graphic data 80 is output to the graphical user interface. Graphic data 80 displays the effects of the first and second adjustable parameters in the PALM procedure that is the basis for data acquisition and data processing. This graphic data 80 displays object plane positions 86 and 87, and further displays a measurement area 88 in which a three-dimensional measurement can be performed and an achievable axial resolution 89. Yes. Such information may be superimposed on the overview image 85 of the object. The calculation device of the microscope system can calculate the measurement region 88 and the achievable resolution 89 according to the values set as the positions of the first and second object planes, respectively. In addition, the computing device may calculate the measurement region 88 and resolution 89 in response to additional information that may be input by the user, such as the density of fluorescent molecules in the sample.

  The graphic data 80 may be generated according to values respectively set for the first adjustable parameter and the second adjustable parameter. If one of these values is changed, i.e. the position of one of the object planes is changed, the graphic data 80 may be adapted accordingly.

  The graphic data generated and output to describe the data acquisition or data processing of the PALM procedure has been described with reference to FIGS. 8-11 in connection with some examples, but additional adjustable parameters. Graphic data can be selectively output. Yet another example of such an adjustable parameter is a setting that specifies whether molecular images acquired at different object planes are grouped before or after locating individual molecules. For these two different cases, graphic data indicating the difference in data processing may be generated and output. Further, for each of the two different cases, a preview of the resulting data for each of those settings may also be generated and output.

  12 and 13 schematically show a graphical user interface that can be output to the light output device 3 of the microscope system. The graphical user interface includes graphic data that displays the effects of data filtering. The microscope system may be configured for microscopy according to SIM procedures, PALM procedures, or other procedures in which raw data filtering is performed. As soon as the user requests information about data filtering and information about the effects of data filtering in the procedure in which the data processing is performed, graphical data as schematically shown in FIGS. 12 and 13 is output. May be.

  A data array 91 is displayed on the graphical user interface. The pixel values in this array are displayed with different brightness levels and / or different values, which may be shown in different fields of the data array 91. Also displayed in the graphical user interface is a filter kernel 92 having typical filter coefficients. This filter coefficient may be determined according to the value of the filter currently set for data processing.

  As schematically illustrated in FIG. 13, the user may move the filter kernel 92 to various positions in the data array 91. In response, the effect of the filter kernel at various positions in the data array is calculated and output to the graphical user interface. As schematically shown in FIG. 13, numerical values obtained as a result of filtering the data array 91 by the filter kernel 92 may be determined and output according to the position where the filter kernel 92 is arranged. The results of filtering using the filter kernel 92 may be encoded in various alternative or additional ways. As an illustrative example, filtering results may be output using different grayscale levels, different colors, or the like to display the effect of the filter in the data array 91.

  The data array 91 may be a typical data array. If the raw data has already been acquired, the data array 91 may be a part of the acquired raw data. The user may then selectively position the filter kernel in the portion of the raw data that is of interest to the user. In response, the user will directly understand the effect of filtering on that particular portion of the raw data. In this way, various filter characteristics may be tested. Thereafter, data processing may be performed using a filter kernel set in a user-defined manner.

  14 and 15 schematically show a graphical user interface output to the light output device 3 of the microscope system. The graphical user interface includes graphic data that displays the separation of multi-channel images. The microscope system may be configured to implement such a method that multi-channel image separation is performed in its data processing stage. When the user requests information about channel separation, graphic data as schematically shown in FIGS. 14 and 15 may be output. This helps the user to understand how changing the number of channels or selecting a data weighting option in response to data uncertainty affects data processing.

  The graphical user interface has a display area 93 for displaying the value of the adjustable parameter. The graphical user interface has an adjustment element 94 that allows the user to adjust the settings of the adjustable parameters. As an illustrative example, the tunable parameter may be the number of channels, i.e. it may represent the total number of different channels. The graphical user interface further includes an adjustment element 95 for selecting a weighting option.

  Graphic data 96 is output to the graphical user interface. Graphic data 96 displays how the number of channels and weighting options affect data processing. Graphic data 96 may be adapted in response to user input that changes the selection of channel number or weighting options. As an illustrative example, when the user selects a weighting option, the graphic data 96 may be adapted accordingly to reflect the user's selection.

  From a technical point of view, the separation of multi-channel images can be compared to solving an overdetermined equation system. The solution may be obtained, for example, using the least square method. In order to visualize the separation of the channels, graphical data may be generated and output that compares the solution of the system of equations to identifying an average line through multiple measurement points. The intersection of the average straight lines corresponds to the solution of the equation system. FIG. 14 schematically illustrates graphical data 96 including a plurality of measurement data 97 and an associated average line. The uncertainty of the solution can be illustrated as a circle 98 around the mean line intersection. Here, the channel separation of the multi-channel image as a solution of the overdetermined equation system is expressed as a straight line that intersects in two dimensions. This visualizes the operating principle of the underlying algorithm in an n-dimensional data space where channel separation is actually performed.

  FIG. 15 shows how the graphic data 96 is adapted when the weighting option is selected. In order to visualize the effect of the weighting option, an error bar 99 may be additionally displayed. Furthermore, the weighting function may cause a shift of the solution and / or change the uncertainty of the displayed solution. This can also be reflected in the graphic data 96.

  According to various embodiments, graphic data can be output to a light output device of a microscope system to display optical transitions of molecules. The displayed optical transition may be an optical transition that is excited in data acquisition. Examples where such graphical data may be useful include a microscope system for luminescence microscopy, such as a microscope system configured to perform microscopy according to the PALM procedure, for example. . The calculation device of the microscope system calculates the occupancy probability and / or transition speed between molecular states of the photoactivatable or photoswitchable molecule according to the value of the adjustable parameter set by the user, respectively. can do. As an illustrative example, occupancy probabilities and / or transition rates between molecular states may be calculated for dye molecules. The occupation probability and / or the transition speed can be output as graphic data to the light output device. The output graphic data may include a display of a Yabronsky diagram or a Markov diagram. The graphic data including such a figure may be adjusted when the power density or wavelength value of the switching signal used for switching the dye molecules is adjusted. Also, the graphic data including such a figure can also be adapted when the power density or wavelength of the excitation signal used for luminescence excitation of the dye molecules is adjusted. Such an implementation method will be described in more detail with reference to FIGS.

  16 and 17 schematically illustrate a graphical user interface. This graphical user interface can be output to the light output device 3 of the microscope system. This graphical user interface may include graphical data that visualizes the effect of tunable parameters on the occupancy probability of a molecular state and / or the transition rate between molecular states. The microscope system may be configured to perform luminescence microscopy in which molecular photoswitching or photoactivation is performed for raw data acquisition. For example, when the user requests information about how a change in laser power density or a change in laser spectrum affects a detectable emission signal, the figures include those shown in FIGS. Graphic data may be output.

  The graphical user interface has a display area 101 for displaying the value of the first adjustable parameter. The graphical user interface has an adjustment element 102 that allows the user to adjust the value of the first adjustable parameter. The graphical user interface has a display area 103 for displaying the setting value of the second adjustable parameter. The graphical user interface has an adjustment element 104 that allows the user to adjust the setting of the second adjustable parameter. As an illustrative example, the first adjustable parameter may correspond to the power density of the laser. The second tunable parameter may correspond to the spectrum of the laser, or the wavelength at which the laser spectrum is maximized, where the laser is used for molecular switching and / or activation.

  Graphic data 105 is output to the graphical user interface. Graphic data 105 displays the effect of laser power and / or laser spectrum on the molecular process triggered during data acquisition. Graphic data 105 may graphically display different molecular states. As an illustrative example, two singlet states and two triplet states may be displayed as shown in FIGS. Furthermore, an arrow representing a transition between molecular states is displayed.

  The calculation device of the microscope system is configured to calculate the occupation probability of different molecular states according to the set values of the first and second adjustable parameters. Alternatively or additionally, the computing device may be configured to calculate transition rates between these molecular states in response to values set for the first and second adjustable parameters, respectively. The calculation device may determine the occupation probability and / or the transition speed according to the photoactive molecule used in the sample. The molecule may be a specific dye molecule. Some implementations may allow the user to enter information about the photoactive molecule or dye via the user interface. In other implementations, the computing device may be configured to control the microscope system such that a spectral scan is achieved with coarse spectral resolution at the initial data acquisition. The computing device may then automatically determine the dye being used by comparing the acquired spectrum with data stored in a database.

  To determine the transition rate, the computing device may query the database. The database may include transition rate values as a function of a plurality of discrete values of the first and second adjustable parameters. As an example for purposes of illustration, these transition rates may be stored in a database for multiple laser powers and multiple laser spectra. The computing device may be configured to interpolate or extrapolate the transition speed stored in the database, depending on the power and wavelength settings set by the user for the switching signal or excitation signal. Based on the transition speed, the occupation probability can be calculated by the calculation device. The occupancy probability can be determined according to the transition speed using a mathematical method known in the Markov process.

  Alternatively, the occupation probability and / or transition speed may be calculated using a model. To determine the occupancy probability and / or transition rate, the computing device may retrieve the relevant molecular parameters from the database. Examples of relevant molecular parameters include transition matrix elements for optical transitions, relaxation rates for natural relaxation, and other transition probabilities. The transition speed is calculated according to the value of the adjustable parameter set by the user. The model used to calculate the transition rate may depend on additional parameters such as temperature. The computing device may be connected to a sensor of the microscope to automatically obtain information about the current temperature value, the pH value of the sample, or other environmental parameters. The computing device may use the outputs of these sensors in determining the occupation probability and / or transition rate. Alternatively or additionally, the computing device may allow the user to enter a temperature value or pH value, thereby providing the user with information indicating the effect of the temperature or pH value in the optical process. You can also. The occupancy probability may be calculated using the transition rate, for example based on a technique developed for Markov processes.

  In the implementation method shown in FIGS. 16 and 17, the calculation device controls the light output device so that the occupation probability calculated by the calculation device is visualized on the light output device 3. The computing device may generate the graphic data 105 such that the size or color coding of the bar symbols 106-109 is proportional to the occupation probability. Bar symbols 106-109 are assigned to relevant molecular states of molecules that can be photoactivated or switched. In addition or alternatively, information on the transition speed may be output. The computing device may generate the graphic data 105 so that the thickness of the transition arrows 110 and 111 is proportional to the transition speed. Alternatively or additionally, the occupation probability and / or the transition speed may be displayed in the graphic data 105 using alphanumeric symbols.

  As shown in FIG. 17, the computing device may adapt the graphic data 105 according to the values respectively set for the adjustable parameters by the user. If the user increases the laser power setting, the graphic data 105 may be adjusted to represent a new occupancy probability and / or transition rate.

  The graphical data representing the probability of occupancy may be adjusted over time to visualize dynamic processes in which the population of individual molecular states increases or decreases. Such dynamics are obtained as a result after the switching signal or activation signal is applied to the photoactive molecule. This allows the user to intuitively understand the processes that are the basis for data acquisition and data processing.

  The information about the occupation probability and / or the transition speed may be output not only in the form of a Yabronsky diagram but also in the form of a Markov diagram. This will be described in more detail with reference to FIGS.

  FIG. 18 shows graphic data 115 output to the light output device of the microscope system. The graphic data 115 is generated by a calculation device of the microscope system. Graphic data 115 includes a Markov diagram in which the various associated states of the dye molecule are represented by alphanumeric symbols 116-118. The relevant state of the dye molecule may be automatically determined by the computing device as described with reference to FIGS. The state of the associated molecule may still change as the power density or spectrum of the signal used to excite or switch the molecule changes. The transition speed determined by the computing device can be output as a numerical value. These numbers are displayed by boxes 119, 120, which may be incorporated into the Markov diagram. The transition speed may be determined as described with reference to FIGS.

  Additional information may be output to further assist the user in understanding the procedure underlying the data acquisition. As an illustrative example, the computing device may be configured to calculate an average period for which the fluorescent state of the dye molecule continues. This period depends on the power density of the excitation signal. The implementation method will be described in more detail with reference to FIG.

  FIG. 19 schematically shows a graphical user interface output to the light output device 3 of the microscope system. The microscope system may be configured to perform luminescence microscopy according to a procedure in which photoswitching or photoactivation of molecules is performed for raw data acquisition.

  The graphical user interface includes graphic data 125. The graphic data 125 includes a Yablonsky diagram or a Markov diagram as described with reference to FIGS. In addition, the graphic data includes data 124 that represents the average duration that the fluorescent state of the dye molecule lasts. The calculation device automatically calculates the average period for which the fluorescence state of the dye molecules continues. The computing device may determine this period according to the previously calculated molecular transition rate. The period during which the fluorescent state of the molecule continues may be visualized by the size or brightness of the output symbol, such as the size or brightness of the circle 124 or bar. A graphical output of the average duration over which the molecule's fluorescent state continues helps the user in determining the exposure time for data acquisition.

  The graphical user interface may include a display area 121 for displaying an exposure time value set by the user. The graphical user interface may further include an adjustment element 122 that allows the user to set the exposure time. The computing device may generate the graphic data 125 to further include a graphic display 123 of the set exposure time. As shown in FIG. 19, the graphic display 124 of the average period in which the fluorescence state of the molecule continues and the graphic display 123 of the set exposure time allow the user to easily compare the two times. , May be generated.

  As described with reference to FIGS. 3-19, information about the effects of various adjustable parameters can be output to the user via a graphical user interface. This provides the user with information about the importance of various adjustable parameters for data acquisition or data processing. This helps the user in the process of identifying appropriate values for various tunable parameters that can be used for data acquisition and / or data processing. As soon as the set values of the various adjustable parameters are determined by the user, data acquisition and / or data processing can be performed using the values set by the user definition.

  The embodiment described with reference to the drawings may be modified in another embodiment. As an illustrative example, graphical data different from that shown in FIGS. 3-19 and described with reference to these figures may be output in yet another embodiment.

  In some embodiments, the process flow can be illustrated based on a particular model. A flowchart representation of the process flow may be generated and output. The flowchart representation may be generated or adapted depending on which one of the various adjustable parameters the user has selected.

  In some embodiments, various steps of data acquisition or data processing may be visualized using animation.

  In other embodiments, highlighting one or several terms of the formula displayed in the graphic data, thereby explaining the importance of the adjustable parameters for the various terms of the formula Can do.

  In some embodiments, simulation results may be output. As an illustrative example, the principle of operation of a filter such as a cut-off filter can be visualized by outputting simulation data.

  In some embodiments, pigment properties can be visualized. Data regarding the dyes contained in the sample can be included in the user input. Alternatively or additionally, the microscope system may be configured to automatically identify the dye used in the sample. The spectrum of the optical signal used for dye molecule switching or excitation may be graphically displayed, thereby assisting the user in identifying settings suitable for the switching and excitation signals depending on the dye characteristics. When a wavelength is selected, information about the dye suitable for use at that wavelength may be output.

  As described with reference to FIGS. 16-19, information indicating the effects of different switching or excitation wavelengths used in the PALM procedure may be output as graphic data. Also, graphic data that displays the influence of the power of the switching or excitation signal of the dye used in the sample may be output. The computing device may visualize the calculated transition speed graphically by calculating the transition speed and outputting a Yabronsky diagram, Markov diagram or other graphical representation. The figure may be calculated according to the laser power, the wavelength of the switching signal, and the dye characteristics. Alternatively or additionally, information about the photobleaching or photodamage of the sample may be output.

  In some embodiments, the effect of changing the value set for the adjustable parameter may be visualized by performing a simulation using the acquired raw data. A preview image may be generated and output by a graphical user interface.

  In some embodiments, a data processing procedure that includes multiple steps may be recorded such that intermediate results of the data processing are stored in the form of image data. For this purpose, various versions of the image data can be represented graphically. These different versions can be the resulting version after the raw data has been processed with a different number of steps in the data processing procedure. As a result, a data processing protocol is realized. This protocol may be generated using the raw data actually acquired. Alternatively, typical data stored in a storage medium of a library or computing device may be used as input data for establishing this protocol.

  In each of the various embodiments described herein, graphic data may be output at the microscope system while the user is setting values for various adjustable parameters. This helps the user plan data acquisition or data processing.

  The output of graphic data may be repeated in time series for each of the various embodiments described herein. As an illustrative example, graphic data associated with the first adjustable parameter may be output first. Thereafter, output of graphic data associated with the second adjustable parameter may be performed. In this way, the user is provided with information about the importance of the various adjustable parameters with respect to the procedure underlying the data acquisition or data processing.

  Although exemplary embodiments have been described in connection with PALM or SIM procedures, embodiments of the present invention are not limited to such microscopy techniques. The microscope system and the microscope method according to the embodiment can be widely used in a microscope technique in which a plurality of adjustable parameters are set in a user-defined manner.

  While embodiments of the present invention have been described with reference to the above embodiments in connection with specific microscopy techniques, it will be appreciated that other embodiments are possible and are not described herein. The embodiments illustrated and described in can be variously modified without departing from the scope of the invention as defined by the appended claims. Upon reading and understanding the above detailed description, various modifications and changes will occur. Therefore, it should be clearly understood that the above description is for illustrative purposes only and is not limiting. Thus, the present invention is intended to be construed as including all such modifications and variations as long as they fall within the scope of the appended claims and their equivalents.

DESCRIPTION OF SYMBOLS 1 Microscope system 2 Computation apparatus 3 Optical output apparatus 4 Input apparatus 5 Radiation source 6 Lens 7 Dichroic beam splitter 8 Objective lens 9 Filter 10 Tube lens 11 Detector 12, 13 Laser 14 Beam combiner 15 Optical filter 16 Graphic data 19 Sample (object)
21 Microscope system 22 Illumination device 23 Pattern generator 24 Lens 25 Sample holder 26 Imaging optical system 31 Display area 32 Adjustment element 33 Display area 34 Adjustment element 35 Graphic data 36 Line 37 Graphic display 38 Line 40 Line 41 Display area 42 Adjustment Element 43 Graphic data 44 Preview image 45 Graphic data 46 to 56 Block 61 Display area 62 Adjustment element 63 Graphic data 64 Camera signal 65 Fit mask display 66 Camera signal 71 Objective lens 72, 73 Object plane 74 Inset 75, 76 Two-dimensional image 80 Graphic data 81 Display area 82 Adjustment element 83 Display area 84 Adjustment element 85 Overview image 86, 87 Position 88 of object plane Fixed area 89 Resolution 91 Data array 92 Filter kernel 93 Display area 94, 95 Adjustment element 96 Graphic data 97 Measurement data 98 Yen 99 Error bar 102 Adjustment element 103 Display area 104 Adjustment element 105 Graphic data 106-109 Bar symbol 110 , 111 Transition arrow 115 Graphic data 116 to 118 Alphanumeric code 119, 120 Box 121 Display area 122 Adjustment element 123 Graphic display 124 Circle 125 Graphic data

Claims (15)

A microscope system,
A microscope for data acquisition (5-11; 11, 22-26) having at least one controllable component (5; 23);
Control during data acquisition of the at least one controllable component (5; 23) of the microscope (5-11; 11, 22-26) and / or the microscope (5-11; 11, 22-26) A computing device (2) configured to perform data processing of the raw data acquired by
An optical output device (3) connected to the computing device (2),
The microscope (5-11; 11, 22-26) and the calculation device (2) are configured to execute the data acquisition and / or the data processing based on values respectively set for a plurality of adjustable parameters. Configured,
The computing device (2) is connected via the light output device (3) to graphic data (16; 35; 43; 45; 63; 80; 91, 92; 96; 105; 115; 125) is selectively output, and the adjustable parameter is selected in a user-defined form from the plurality of adjustable parameters and the output graphic data (16; 35; 43; 45; 63; 80; 91, 92; 96; 105; 115; 125) are assigned to the selected tunable parameters and are the basis for the data acquisition and / or the data processing. A microscope system representing the influence of the selected adjustable parameter in at least one step of the procedure. The calculation device (2) is configured to calculate image data from a plurality of two-dimensional frames collected in the data acquisition in the data processing,
The computing device (2), for the at least one adjustable parameter affecting the data processing, via the light output device (3), the graphic data (16; 35; 43; 45; 63; 80). 91, 92; 96; 105; 115; 125), and the output graphic data (16; 43; 45; 63; 91, 92; 96; 105; 115; 125). The microscope system according to claim 1, wherein) represents the influence of the selected adjustable parameter in the procedure on which the data processing is based. An input device (4; 32, 34; 42; 62) connected to the computing device (2) for receiving user input for adjusting a value of a given adjustable parameter selected from the plurality of adjustable parameters; 82, 84; 94; 102, 104; 122),
The computing device (2) automatically uses the graphic data (16; 35; 43; 45; 63; 80; 91, 92; 96; 105; 115; 125) assigned to the given adjustable parameter. The microscope system according to claim 1, wherein the microscope system is configured to output to the microscope.   The computing device (2) is responsive to the value set for the selected adjustable parameter for the graphic data (16; 35; 43; 45; 63; 80; 91, 92; 96; 105; 115). 125) The microscope system according to any one of claims 1 to 3, wherein the microscope system is configured to output 125).   The computing device (2) is responsive to the values set for further adjustable parameters, depending on the graphic data (16; 35; 43; 45; 63; 80; 91, 92; 96; 105; 115; The microscope system of claim 4, wherein the second adjustable parameter is different from the selected adjustable parameter.   The computing device (2) chronologically stores the graphic data (16; 35; 43; 45; 63; 80; 91, 92; 96; 105; 115; 125) respectively assigned to different adjustable parameters. The microscope system according to any one of claims 1 to 5, wherein the microscope system is configured to output to the microscope.   The computing device (2) is configured to execute the graphic data (16; 35; 43; 45; 63; 80; 91, 92; 96; 105; 115) before the data acquisition and / or the data processing is performed. 125) The microscope system according to any one of claims 1 to 6, wherein the microscope system is configured to output 125). The microscope system (1) is configured to perform the data acquisition and / or the data processing using photoactivatable or photoswitchable molecules,
The computing device automatically calculates the population probability of the molecular state of the molecule and / or the transition speed between molecular states, and the graphic data (105; 115; 125) is the molecule of the molecule. Configured to output the graphic data (105; 115; 125) to represent the influence of the selected adjustable parameter on the population probability of states and / or the transition rate between the molecular states. The microscope system according to claim 7, wherein the computing device is configured to output the graphic data (105; 115; 125) before the data acquisition is performed. The computing device (2) is configured to output the graphic data (16; 35; 43; 45; 63; 80; 91, 92; 96) after the data acquisition is performed;
The computing device (2) via the light output device (3), the graphic data (16; 35; 43; 45; 63; 80; 91, 92) assigned to the selected adjustable parameter. 96) in addition to being configured to output further graphic information (44), said further graphic information (44) being obtained by said microscope (5-11; 11, 22-26) in said data acquisition. The microscope system according to claim 1, wherein the microscope system is generated according to the collected raw data. The microscope system (21) is configured to perform the data acquisition using structured illumination,
At least one of the plurality of adjustable parameters selectable for outputting the graphic data (16; 35; 43; 45; 91, 92; 96) is a periodicity of an illumination pattern, The number of different orientations of the illumination pattern and a filter function for filtering the raw data acquired by the microscope (5-11; 11, 22-26). The microscope system according to any one of 9. The microscope system (1) is configured to perform the data acquisition and / or data processing using photoactivatable or photoswitchable molecules,
At least one of the plurality of adjustable parameters that is selectable for outputting the graphic data (16; 35; 43; 45; 91, 92; 96) is a mask for the fitting step. Size, spectrum of a switching signal for switching of the molecule, intensity of the switching signal for switching of the molecule, spectrum of an activation signal for activation of the molecule, and for activation of the molecule The microscope system according to any one of claims 1 to 10, wherein the microscope system is selected from the group consisting of intensity of activation signals. The microscope system (1) is configured to perform the data acquisition and / or data processing using photoactivatable or photoswitchable molecules,
The microscope (5-11) is configured to perform the data acquisition for a plurality of object planes (72, 73), the plurality of object planes (72, 73) being spaced apart from each other. And
The microscope (5-11) is controllable to adjust the position of the object plane (72, 73);
The computing device (2) is adapted to display the graphic data (2) representing the influence of the position of the object plane (72, 73) on the size of the measurement area (88) and / or the achievable resolution (89). The microscope system according to any one of claims 1 to 11, wherein the microscope system is configured to output (80). Microscopy,
The microscopy includes obtaining raw data and processing the obtained raw data in a procedure comprising a plurality of steps, the obtaining the raw data and / or the obtaining of the obtained raw data. Processing is performed according to the values set for multiple adjustable parameters,
The microscopy is based on graphic data (16; 35; 43; 45; 63; 80; 91, 92; 96; 105; 115; 125) via the light output device (3), depending on the adjustable parameters. The adjustable parameter is selected from the plurality of adjustable parameters in a user-defined form, and the output graphic data (16; 35; 43; 45; 63; 80; 91, 92; 96; 105; 115; 125) are assigned to the selected adjustable parameter, and the acquisition of the raw data and / or the processing of the acquired raw data. A microscopy showing the influence of the selected tunable parameter on at least one step of the procedure on which it is based.   14. Microscopy according to claim 13, wherein the microscopy is carried out using the microscope system (1; 21) according to any one of claims 1-12. A non-transitory storage medium for storing instruction codes,
15. The instruction code comprises computer-executable instructions, which are executed by the computing device (2) of the microscope system (1; 21) to perform the microscopy method according to claim 13 or claim 14. A storage medium for instructing the microscope system (1; 21) to perform.

Images