KR102628867B1 - System of Structured Illumination Microscopy and Control Method thereof - Google Patents

Abstract

A method of controlling a structured illumination microscope system according to another embodiment of the present invention includes the steps of (A) a control unit generating a DMD pattern; (B) the control unit performing a diffraction operation on diffraction light generated by the DMD pattern; (C) the control unit determining whether the result of the diffraction calculation satisfies three conditions in order to find a DMD pattern that creates a desired structured illumination cycle; (D) the control unit storing the results of the diffraction calculation satisfying the three conditions together with the corresponding DMD pattern as a Look Up Table (LUT); and (E) the control unit extracting a structured lighting pattern of a desired cycle from the stored LUT.

Description

Structured illumination microscopy system and control method thereof {System of Structured Illumination Microscopy and Control Method thereof}

The present invention relates to a structured illumination microscope system and a control method thereof, and particularly to a structured illumination microscope system and a control method that calculates the pattern of a digital micro-reflective indicator at high speed and implements desired illumination.

Structured illumination microscopy is a type of super-resolution microscope that exceeds the resolution limits of conventional optical microscopes by irradiating a sample with structured illumination in the form of a sine wave. Unlike other super-resolution microscopes, it is widely used in various fields because it can image regardless of the type of fluorescent material.

Conventional methods of creating structured lighting use a diffraction grating, a spatial light modulator (SLM), or a digital micromirror device (DMD). Diffraction gratings are inexpensive, but they are not preferred because it is very difficult to change the cycle and direction of structured lighting. SLMs are often used because the alignment method is simpler than DMD, but are expensive and have a slow light modulation speed. There is a downside. On the other hand, when using DMD, alignment is more difficult than SLM, but it is far superior in terms of price and modulation speed.

To create a pattern that floats on a conventional DMD, first select an objective lens that fits the size of the sample and determine the period and direction of structured illumination accordingly. The diffraction pattern created by the laser light to create the determined structured illumination in the Fourier domain of the DMD is calculated, and a mask corresponding to it is created. Finally, the DMD pattern that produces the diffraction result passing through the mask located in the Fourier domain is calculated.

However, in this prior art, when the magnification or numerical aperture (NA) value of the objective lens changes, the period of structured illumination irradiated to the sample must be changed, but since the mask is not variable in response to such changes, it can flexibly cope with such changes. I can't.

In addition, the advantage of DMD is that the period or direction of the desired structured lighting can be controlled electronically, but there is a problem in that the degree of freedom is reduced by the mask used in the Fourier domain.

Patent document: Registered Patent Publication No. 10-1479249

The present invention was created to solve the above problems, and the purpose of the present invention is to provide a structured illumination microscope system that calculates patterns displayed on a DMD at high speed and implements desired structured illumination.

Another object of the present invention is to provide a control method for a structured illumination microscope system that calculates patterns displayed on a DMD at high speed and implements desired structured illumination.

A structured illumination microscope system according to an embodiment of the present invention includes a DMD (Digital Micromirror Device) to which laser light from a laser light source is irradiated; a first lens and a second lens provided in the optical path reflected from the DMD; an aperture provided between the first lens and the second lens; a reflector provided behind the second lens; a dichroic mirror spaced apart from the reflector and provided between the camera and a third lens; an objective lens that condenses the light transmitted through the third lens and irradiates it to the sample on the sample holder; and a control unit connected to the DMD, the iris, and the camera.

In the structured illumination microscope system according to an embodiment of the present invention, the aperture includes a housing of the aperture and a light blocking band provided across the central portion of the opening formed in the housing of the aperture, and the light blocking band includes a central diffraction light ( It is made of an opaque material that blocks S0) and is characterized by having a width provided between two adjacent diffraction lights (S1, S2) spaced at the same distance from the central diffraction light (S0).

In the structured illumination microscope system according to an embodiment of the present invention, the aperture includes a housing of the aperture, a light blocking film provided at the center of the opening formed in the housing of the aperture, and three wires connected between the light blocking film and the housing of the aperture. Including,

The light blocking film is disk-shaped and formed of an opaque material that blocks the central diffraction light (S0), and is formed between two adjacent diffraction lights (S1 and S2) spaced at the same distance from the central diffraction light (S0). It is characterized by having a diameter provided.

In the structured illumination microscope system according to an embodiment of the present invention, the control unit adjusts the size of the opening of the aperture according to the magnification and numerical aperture of the objective lens to meet the resolution limit.

Alternatively, a method of controlling a structured illumination microscope system according to another embodiment of the present invention includes the steps of (A) the control unit generating a DMD pattern; (B) the control unit performing a diffraction operation on diffraction light generated by the DMD pattern; (C) the control unit determining whether the result of the diffraction calculation satisfies three conditions in order to find a DMD pattern that creates a desired structured illumination cycle; (D) the control unit storing the results of the diffraction calculation satisfying the three conditions together with the corresponding DMD pattern as a Look Up Table (LUT); and (E) the control unit extracting a structured lighting pattern of a desired period from the stored LUT.

In the method for controlling a structured illumination microscope system according to another embodiment of the present invention, step (B) includes (B-1) representing the DMD pattern as a vector and approximately expressing it in the form of an approximation function; and (B-2) performing a two-dimensional Fourier transform on the approximate function form.

The control method of the structured illumination microscope system according to another embodiment of the present invention is based on the above three conditions in step (C): ① the light (S1, S2) diffracted at the closest distance from the central diffracted light (S0) is 2 ②Whether the distance difference (R2-R1) between the closest light (S1) and the second closest light (S3) from the central diffracted light (S0) has a value greater than a certain value, and ③Pixel in the DMD pattern It is characterized by including whether the phase change that occurs when one square is pushed has a value other than π/2×n (n=0,1,2,3,...).

In the method for controlling a structured illumination microscope system according to another embodiment of the present invention, step (E) includes (E-1) converting the stored LUT by reflecting an orthogonal projection in the space domain; (E-2) arbitrarily setting a boundary range according to the desired cycle of structural lighting; (E-3) deleting only the diffraction light pattern in the boundary range; and (E-4) selecting a diffraction light pattern having an arbitrary angle with respect to the remaining diffraction light pattern.

The features and advantages of the present invention will become more apparent from the following detailed description based on the accompanying drawings.

Prior to this, terms or words used in this specification and claims should not be interpreted in their usual, dictionary meaning, and the inventor should appropriately define the concept of the term in order to explain his or her invention in the best way. It must be interpreted with meaning and concept consistent with the technical idea of the present invention based on the principle that it can be done.

The structured illumination microscope system and control method according to an embodiment of the present invention has the effect of calculating the DMD pattern displayed on the DMD at high speed, easily extracting structured illumination of the desired cycle, and analyzing samples by selecting various objective lenses. there is.

1 is a configuration diagram of a structured illumination microscope system according to an embodiment of the present invention.
Figure 2a is a front view of an aperture constituting a structured illumination microscope system according to an embodiment of the present invention.
Figure 2b is a rear view of an aperture constituting a structured illumination microscope system according to an embodiment of the present invention.
Figure 3a is a front view of another aperture constituting a structured illumination microscope system according to an embodiment of the present invention.
Figure 3b is a rear view of another aperture constituting a structured illumination microscope system according to an embodiment of the present invention.
Figure 4 is a flowchart illustrating a control method of a structured illumination microscope system according to another embodiment of the present invention.
Figure 5 is an exemplary diagram showing the process of generating a DMD pattern according to a control method of a structured illumination microscope system according to another embodiment of the present invention.
Figure 6 is an example diagram illustrating three conditions for saving as a LUT according to a control method of a structured illumination microscope system according to another embodiment of the present invention.
Figure 7 is an exemplary diagram showing the process of extracting structured illumination according to a control method of a structured illumination microscope system according to another embodiment of the present invention.
Figure 8 is an example diagram for explaining the process of implementing structured illumination based on the resolution limit line of the objective lens according to the control method of the structured illumination microscope system according to another embodiment of the present invention.

The objectives, specific advantages and novel features of the present invention will become more apparent from the following detailed description and preferred embodiments taken in conjunction with the accompanying drawings. In this specification, when adding reference numbers to components in each drawing, it should be noted that identical components are given the same number as much as possible even if they are shown in different drawings. Additionally, terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. Additionally, in describing the present invention, if it is determined that a detailed description of related known technologies may unnecessarily obscure the gist of the present invention, the detailed description will be omitted.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings. Figure 1 is a configuration diagram of a structured illumination microscope system according to an embodiment of the present invention, Figure 2a is a front view of an aperture constituting the structured illumination microscope system according to an embodiment of the present invention, and Figure 2b is an embodiment of the present invention. FIG. 3A is a rear view of an aperture constituting a structured illumination microscope system according to an embodiment of the present invention, and FIG. 3A is a front view of another aperture constituting a structured illumination microscope system according to an embodiment of the present invention, and FIG. 3B is an embodiment of the present invention. This is a rear view of another aperture that constitutes a structured illumination microscope system according to . The description of the structured illumination microscope system according to an embodiment of the present invention is the same as that of the conventional structured illumination microscope system and will be omitted.

The structured illumination microscope system 100 according to an embodiment of the present invention includes a laser light source unit 101, a digital micromirror device (DMD) 102 that generates a periodic pattern, and an agent provided in the light path reflected from the DMD 102. 1 lens 103 and the second lens 105, an aperture 104 provided between the first lens 103 and the second lens 105, and a reflector 106 provided behind the second lens 105. , a dichroic mirror (107) spaced apart from the reflector 106 and provided between the camera 108 and the third lens 109, condenses the light passing through the third lens 109 to produce a sample It includes a control unit (not shown) connected to an objective lens 110 that irradiates the sample on the holder 120, a DMD 102, an aperture 104, a camera 108, etc.

As shown in FIG. 1, the DMD 102 is a device equipped with a plurality of micro mirrors (hereinafter referred to as pixels) in a two-dimensional array pattern under the control of a control unit, and emits irradiated light from the laser light source unit 101. It is a device that transmits spatial modulation while reflecting to the first lens 103. At this time, the DMD 102 can generate a two-dimensional array pattern in which a plurality of pixels have a repeated on and off pattern with a predetermined period under the control of the controller.

The aperture 104 is a member that filters the diffracted light that has passed through the first lens 103. The diameter of the internal opening is set according to the magnification and numerical aperture of the objective lens 110 under the control of the control unit, and the diffracted light in the center is set. It performs the function of blocking.

Specifically, as shown in FIG. 2, the aperture 104 includes a housing 104-1 of the aperture and a light blocking band 104-2 provided across the central portion of the internal opening. The light blocking zone 104-2 of this aperture 104 is formed of an opaque material that blocks the central diffraction light S0 and is formed of two adjacent diffraction lights S1 spaced apart from the central diffraction light S0 at the same distance. ,S2) It is installed with a width provided between.

Alternatively, as shown in FIG. 3, the aperture 204 may be in the form of an aperture housing 204-1, a light blocking film 204-2 provided at the center of the internal opening, and a light blocking film 204-2 and the housing. It is provided with three wires (204-3) connected between (204-1). The light blocking film 204-2 of this other aperture 204 is, for example, disk-shaped and is formed of an opaque material that blocks the central diffraction light S0, and is adjacent to the central diffraction light S0 at an equal distance from the central diffraction light S0. It is installed with a diameter provided between two diffracted lights (S1, S2).

The control unit is connected to the DMD (102), the aperture (104,204), and the camera (108) to control the overall operation of the structured illumination microscope system (100), in particular, the aperture (104,204) according to the magnification and numerical aperture of the objective lens (110). The sample can be analyzed by setting the diameter of the internal opening, calculating the pattern to be displayed on the DMD 102, and generating structured illumination with a desired period and direction.

Hereinafter, a control method of a structured illumination microscope system according to another embodiment of the present invention will be described with reference to FIGS. 4 to 8. Figure 4 is a flowchart for explaining a control method of a structured illumination microscope system according to another embodiment of the present invention, and Figure 5 is a flow chart for generating a DMD pattern according to a control method of a structured illumination microscope system according to another embodiment of the present invention. It is an example diagram showing the process, and Figure 6 is an example diagram illustrating three conditions for saving as a LUT according to a control method of a structured illumination microscope system according to another embodiment of the present invention, and Figure 7 is another example of the present invention. These are illustrative diagrams showing the process of extracting structured illumination according to a control method of a structured illumination microscope system according to an embodiment, and Figure 8 shows the resolution limit line of the objective lens according to a control method of a structured illumination microscope system according to another embodiment of the present invention. This is an example diagram to explain the process of implementing structural lighting based on .

In the control method of a structured illumination microscope system according to another embodiment of the present invention, as shown in FIG. 4, the control unit first generates a DMD pattern with a certain period using simulation (S410).

Specifically, as shown in FIG. 5, the x-axis and y-axis can be set as the arrangement directions of pixels in the DMD 102, and a DMD pattern having a certain period in the x-axis and y-axis can be generated through simulation.

For the DMD pattern generated in this way, the control unit performs a diffraction calculation on the diffracted light generated by the DMD pattern (S420).

Since the angle or position of the diffracted light generated by the DMD pattern is affected only by the periodicity of the DMD pattern, the control unit can represent the DMD pattern with a certain period on the x-axis and y-axis as a vector and approximate it in the form of an approximation function.

For example, a DMD pattern with a 10-pixel period on the x-axis and a 5-pixel period on the y-axis can be approximated using the Dirac comb function as [Equation 1] below.

Here, white indicates the on state, black indicates the off state, and d indicates the length of one side of the pixel. Additionally, 1 in each vector x _1,2,3,4,5 means on state, and 0 means off state.

Afterwards, the control unit can obtain the result of light diffracted by the DMD pattern in the Fourier domain corresponding to the aperture 104 using the two-dimensional Fourier transform using [Equation 2] below.

Here, u and v are the spatial frequencies corresponding to x and y, respectively.

At this time, if [Equation 2], which is the result of a DMD pattern with a 10x5 cycle, is expanded to the result of a general NxM cycle pattern (N pixel cycle on the x-axis, M pixel cycle on the y-axis), [Equation 3] below It can be expressed as

Using [Equation 3], the diffraction results of light that occur after generating a DMD pattern of a specific period on the DMD can be calculated at high speed without the Fourier transform process.

This [Equation 3] is a result in the spatial frequency domain, and the result in the spatial frequency domain can be converted to the space domain or the diffraction angle domain of light.

In other words, it can be converted into the relationship between the spatial frequency domain and the space domain using [Equation 4] below.

Alternatively, it can be converted into the relationship between the spatial frequency domain and the diffraction angle domain using [Equation 5] below.

At this time, λ is the wavelength of the light source in the laser light source unit 101, and f is the focal length of the first lens 103 that creates the Fourier domain.

After performing the diffraction calculation, the control unit determines whether the diffraction result satisfies three conditions to find a DMD pattern that creates the desired structured illumination cycle (S430).

That is, the three conditions are, referring to FIG. 6, ① there must be only two diffracted lights (S1, S2) at the closest distance from the central diffracted light (S0), and ② there must be only two diffracted lights (S1, S2) at the closest distance from the central diffracted light (S0). The distance difference (R ₂ -R ₁ ) between the closest light (S1) and the second closest light (S3) must be greater than a certain value, and ③ the phase change that occurs when one pixel is moved in the DMD pattern is π/2 × n It must have a value other than (n=0,1,2,3,...).

Specifically, in order to satisfy the condition ① and block the central diffracted light S0, a light blocking band 104-2 or a light blocking film 204-2 is provided at the center of the internal opening in the apertures 104 and 204, The control unit adjusts the size of the openings of the apertures 104 and 204 so that only the two lights S1 and S2 diffracted at the closest distance from the central diffracted light S0 pass through.

The condition of ② is the distance difference based on a specific value set according to the system in order to pass only the two lights (S1, S2) diffracted at the closest distance from the central diffracted light (S0) and block the other lights (S3). The condition is that (R ₂ -R ₁ ) must be greater than a specific value, and the specific value has [Equation 6] below.

Here, d is the spacing of pixels, and D is the diameter of the laser light in the laser light source unit 101.

Since these specific values are dependent on the system, they must be set differently depending on the system situation.

The condition in ③ is a necessary condition for restoring the structured illumination microscope image. If the condition in ③ is not satisfied, order unmixing becomes impossible during the process of restoring the structured illumination microscope image, making restoration of the structured illumination microscope image impossible. do.

The diffraction results that satisfy these three conditions are saved as a Look Up Table (LUT) along with the corresponding DMD pattern (S440).

The above-described process up to the LUT storage step (S440) is performed for each DMD pattern shown in FIG. 5, and the DMD pattern information, including the diffraction results satisfying the three conditions, is stored in a database (not shown) connected to the control unit. It can be stored in the form of a classification table of LUT.

For the LUT stored in this way, the control unit extracts a structured lighting pattern of the desired cycle (S450).

For example, in the laser light source unit 101, the wavelength of the light source is 405 nm, the focal length of the first lens 103 and the second lens 105 is 10 cm, the length of one side of the pixel is 7.56 μm, and the DMD (102) Assume a structured illumination microscope system with a twist angle of 12.6°.

In order to extract a structured illumination pattern when the desired cycle of structured illumination is 25.3 ~ 26.3㎛ in this structured illumination microscope system, the control unit first reflects the orthographic effect on the LUT of the space domain as shown in Figures 7a to 7b. Convert it.

That is, for the LUT of the space domain that displays all the diffracted light that satisfies the three conditions shown in FIG. 7A, the control unit reflects the orthographic effect of the pattern at an angle of 12.6° where the DMD 102 is tilted, as shown in FIG. 7B. It can be converted as shown.

After conversion to reflect the orthographic effect, the control unit arbitrarily sets the boundary range according to the desired period of structural lighting, as shown in FIG. 7C.

That is, the control unit can set the boundary range of the blue circle with a period of 25.3 ㎛ and the red circle with a period of 26.3 ㎛, as shown in FIG. 7C, in response to the case where the desired period of structural lighting is 25.3 ~ 26.3 ㎛.

For this boundary range, as shown in FIG. 7D, the control unit deletes only the diffraction light pattern in the boundary range.

For the remaining diffraction light patterns, the control unit selects diffraction light patterns having arbitrary angles, as shown in FIG. 7E.

When the selected diffraction light patterns are captured by an imaging device at the position of the aperture 104, the control unit can obtain an image as shown in FIG. 7F.

In this way, by using the LUT, it is possible to very quickly find and apply a DMD pattern that creates structured lighting of a desired cycle, and analyze the structured lighting of the desired cycle by examining the sample in the sample holder 120.

In particular, in order to select and apply various objective lenses to the structured illumination microscope system, the control unit refers to different resolution limit lines according to the magnification of the objective lens 110, as shown in FIG. 8, and selects a DMD pattern with a period close to the resolution limit line. The structured lighting generated by extracting from the LUT is examined on the sample.

At this time, the control unit sets the opening sizes of the apertures 104 and 204 to different resolution limits according to the magnification and numerical aperture of the objective lens 110, so that various objective lenses can be used in one structured illumination microscope system.

As such, the control method of the structured illumination microscope system according to another embodiment of the present invention can analyze the sample with high resolution by calculating the DMD pattern displayed on the DMD 102 at high speed and easily extracting structured illumination of the desired cycle.

Although the technical idea of the present invention has been described in detail according to the above preferred embodiments, it should be noted that the above-described embodiments are for illustrative purposes only and are not intended for limitation.

Additionally, an expert in the technical field of the present invention will understand that various implementations are possible within the scope of the technical idea of the present invention.

100: Structured illumination microscope system 101: Laser light source unit
102: DMD 103: first lens
104, 204: Aperture 104-1, 204-1: Housing
104-2: Light blocking zone 105: Second lens
106: Reflector 107: Dichroic Mirror
108: camera 109: third lens
110: Objective lens 120: Sample holder
204-2: Light blocking film 204-3: Wire

Claims (8)

DMD (Digital Micromirror Device) to which the laser light from the laser light source is irradiated;
a first lens and a second lens provided in the optical path reflected from the DMD;
an aperture provided between the first lens and the second lens;
a reflector provided behind the second lens;
a dichroic mirror spaced apart from the reflector and provided between the camera and a third lens;
an objective lens that condenses the light transmitted through the third lens and irradiates it to the sample on the sample holder; and
It includes a control unit connected to the DMD, the aperture, and the camera,
The aperture includes a housing of the aperture and a light blocking band provided across a central portion of an opening formed in the housing of the aperture,
The light blocking zone is formed of an opaque material that blocks the central diffraction light (S0), and has a width provided between two adjacent diffraction lights (S1, S2) spaced at the same distance from the central diffraction light (S0). A structured illumination microscope system characterized by having a.
DMD (Digital Micromirror Device) to which the laser light from the laser light source is irradiated;
a first lens and a second lens provided in the optical path reflected from the DMD;
an aperture provided between the first lens and the second lens;
a reflector provided behind the second lens;
a dichroic mirror spaced apart from the reflector and provided between the camera and a third lens;
an objective lens that condenses the light transmitted through the third lens and irradiates it to the sample on the sample holder; and
It includes a control unit connected to the DMD, the aperture, and the camera,
The aperture includes a housing of the aperture, a light blocking film provided at the center of an opening formed in the housing of the aperture, and three wires connected between the light blocking film and the housing of the aperture,
The light blocking film is disk-shaped and formed of an opaque material that blocks the central diffraction light (S0), and is formed between two adjacent diffraction lights (S1 and S2) spaced at the same distance from the central diffraction light (S0). A structured illumination microscope system characterized by having a diameter provided.
The method of claim 1 or 2,
A structured illumination microscope system, wherein the control unit adjusts the size of the opening of the aperture according to the magnification and numerical aperture of the objective lens to meet the resolution limit.
delete (A) the control unit generates a DMD pattern;
(B) the control unit performing a diffraction operation on diffraction light generated by the DMD pattern;
(C) the control unit determining whether the result of the diffraction calculation satisfies three conditions in order to find a DMD pattern that creates a desired structured illumination cycle;
(D) the control unit storing the results of the diffraction calculation satisfying the three conditions together with the corresponding DMD pattern as a Look Up Table (LUT); and
(E) the control unit extracting a structured lighting pattern of a desired period for the stored LUT;
A control method of a structured illumination microscope system comprising.
According to claim 5,
The step (B) is
(B-1) representing the DMD pattern as a vector and approximately expressing it in the form of an approximation function; and
(B-2) performing a two-dimensional Fourier transform on the approximate function form;
A control method for a structured illumination microscope system comprising:
According to claim 5,
In step (C), the three conditions are ① whether there are only two diffracted lights (S1, S2) at the closest distance from the central diffracted light (S0), ② whether there are only 2 diffracted lights (S1, S2) at the closest distance from the central diffracted light (S0). Whether the distance difference (R2-R1) between the light (S1) and the second closest light (S3) has a value greater than a certain value, and ③ the phase change that occurs when one pixel is pushed in the DMD pattern is π/2×n( A control method for a structured illumination microscope system, including whether it has a value other than n=0,1,2,3,...).
According to claim 5,
The step (E) is
(E-1) converting the stored LUT by reflecting an orthogonal projection in the space domain;
(E-2) arbitrarily setting a boundary range according to the desired cycle of structural lighting;
(E-3) deleting only the diffraction light pattern in the boundary range; and
(E-4) selecting a diffraction light pattern having an arbitrary angle with respect to the remaining diffraction light pattern;
A control method for a structured illumination microscope system comprising: