WO2020179039A1 - Microscope and observation method - Google Patents

Abstract

This microscope (1) is provided with: an illumination optical system (4) which has a scanning unit (18) that scans a sample (S), in a plurality of directions of the sample, with fringe illumination that is a fringe-like illumination generated by light which is for multiple photon excitation of a fluorescent material contained in the sample; a detection optical system (5) on which fluorescent light (L3) generated from the sample by the multiple photon excitation (S) is incident; a detection device (6) having a plurality of detection units (6a) that detect the fluorescent light (L3) from the sample (S) via the detection optical system (5); and an image processing unit (7) that generates an image using the detection results from at least two detection units among the plurality of detection units (6a).

Description

Microscope and observation method

The present invention relates to a microscope and an observation method.

A scanning microscope that detects fluorescence from a sample has been proposed (for example, see Non-Patent Document 1 below).

The microscope according to the first aspect is an illumination including a scanning unit that scans a striped illumination, which is a striped illumination generated by light for multiphoton excitation of a fluorescent substance contained in the sample, in a plurality of directions of the sample. An optical system, a detection optical system in which fluorescence from the sample generated by multiphoton excitation is incident, a detection device having a plurality of detection units for detecting fluorescence from the sample via the detection optical system, and the plurality of It is provided with an image processing unit that generates an image by using the detection results of at least two of the detection units.

The observation method according to the second aspect comprises scanning stripe illumination, which is stripe illumination generated by light for multiphoton exciting a fluorescent substance contained in the sample, in a plurality of directions of the sample, and Detecting fluorescence from the sample generated by photon excitation using a plurality of detection units, and generating an image using detection results of at least two detection units of the plurality of detection units. Including.

It is a figure which shows the microscope which concerns on 1st Embodiment, and the optical path of excitation light. It is a figure which shows the mask and polarizer concerning 1st Embodiment. It is a figure which shows the polarization state of the mask, the polarizer, the interference fringe, and the excitation light which concerns on 1st Embodiment. It is a figure which shows the microscope and the optical path of fluorescence which concern on 1st Embodiment. It is a figure which shows the effective PSF at each position of the detection apparatus which concerns on 1st Embodiment. It is a flowchart which shows the observation method which concerns on 1st Embodiment. It is a figure which shows the process of the image processing part of the microscope which concerns on 2nd Embodiment. It is a flowchart which shows the observation method which concerns on 2nd Embodiment. It is a figure which shows the process of the image processing part of the microscope which concerns on 3rd Embodiment. In 3rd Embodiment, it is a figure which shows the area|region of the frequency space used for component separation. It is a flowchart which shows the observation method which concerns on 3rd Embodiment. It is a flowchart which shows the observation method which concerns on 4th Embodiment. It is a figure which shows the microscope which concerns on 5th Embodiment. It is a figure which shows the modification of the microscope which concerns on 5th Embodiment. It is a figure which shows the microscope which concerns on 6th Embodiment. It is a figure which shows the microscope which concerns on 7th Embodiment. It is a figure which shows the microscope which concerns on 8th Embodiment. It is a figure which shows the mask and polarization state of excitation light which concern on 8th Embodiment. It is a figure which shows the microscope which concerns on 9th Embodiment. It is a figure which shows the polarization state of the excitation light which concerns on 9th Embodiment. It is a figure which shows the microscope which concerns on 10th Embodiment. It is a figure which shows the mask which concerns on 10th Embodiment. It is a figure which shows the microscope which concerns on 11th Embodiment. It is a figure which shows the illumination pupil which concerns on a modification. It is a figure which shows the illumination pupil which concerns on a modification. It is a figure which shows the microscope which concerns on the modification. It is a figure which shows the polarization adjustment part which concerns on the modification. It is a figure which shows the polarization adjustment part which concerns on the modification. It is a figure which shows the microscope which concerns on the modification. It is a figure which shows the microscope which concerns on the modification. It is a figure which shows the phase modulation element which concerns on a modification. It is a figure which shows the positional relationship between the phase modulation element and the excitation light which concerns on a modification. It is a flowchart which shows the observation method which concerns on a modification. It is a flowchart which shows the subflow of the observation method which concerns on a modification.

[First Embodiment]
The first embodiment will be described. FIG. 1 is a diagram showing a microscope and an optical path of excitation light according to the first embodiment. In the following embodiments, the microscope 1 will be described as being a scanning multiphoton excitation fluorescence microscope, but the microscope according to the embodiment is not limited to a scanning microscope or a fluorescence microscope. The microscope 1 includes a stage 2, a light source 3, an illumination optical system 4, a detection optical system 5, a detection device 6, an image processing unit 7, and a control unit 8. The microscope 1 operates as follows.

Stage 2 holds the sample S to be observed. The sample S is a cell or the like that has been fluorescently stained in advance. The sample S contains a fluorescent substance such as a fluorescent dye. The light source 3 emits excitation light L1 that multiphoton excites the fluorescent substance contained in the sample S. Hereinafter, multiphoton excitation may be simply referred to as excitation. The illumination optical system 4 scans the sample S with the interference fringes L2 of the excitation light L1 in a plurality of directions (for example, the X direction and the Y direction). The illumination optical system 4 scans the sample S two-dimensionally with the interference fringe L2. The detection optical system 5 is arranged at a position where the fluorescence L3 (shown later in FIG. 4) generated by the multiphoton excitation from the sample S enters. The detection device 6 includes a plurality of detection units 6a (later shown in FIG. 4) that detect the fluorescence L3 from the sample S via the detection optical system 5. The image processing unit 7 uses the detection results of the two or more detection units 6a of the detection device 6 to generate an image (for example, a super-resolution image). The control unit 8 controls the light source 3, the drive unit 22, the image processing unit 7, and the like. A control device 8C is configured to include the image processing unit 7 and the control unit 8. Hereinafter, each part of the microscope 1 will be described.

The light source 3 includes a light source such as a laser element. The light source 3 generates coherent light in a predetermined wavelength band. The predetermined wavelength band is set to a wavelength band in which the sample S containing a fluorescent substance can be excited by multiple photons. For example, when the excitation wavelength of the sample S is near λ _ex and the sample S is excited by n (n is an integer of 2 or more) photon excitation, the predetermined wavelength band is set to a wavelength band near n×λ _ex. It is desirable to be done. The excitation light L1 is emitted from the light source 3 as, for example, pulsed light. The excitation light L1 emitted from the light source 3 is, for example, linearly polarized light. A light guide member such as an optical fiber 11 is connected to the emission port of the light source 3. The microscope 1 does not have to include the light source 3, and the light source 3 may be provided separately from the microscope 1. For example, the light source 3 may be replaceably (attachable or detachable) provided to the microscope 1. The light source 3 may be externally attached to the microscope 1 during observation with the microscope 1.

The illumination optical system 4 is arranged at a position where the excitation light L1 from the light source 3 is incident. Excitation light L1 is incident on the illumination optical system 4 from the light source 3 through the optical fiber 11. The optical fiber 11 may be a part of the illumination optical system 4 or a part of a light source device including the light source 3. The illumination optical system 4 includes a collimator lens 12, a λ/4 wave plate 13, a polarizer 14, a mask 15 (aperture member), a dichroic mirror 16, a relay optical system 17, and a scan in order from the light source 3 side to the sample S side. The unit 18, the lens 19, the lens 20, and the objective lens 21 are provided.

In the following explanation, the XYZ Cartesian coordinate system shown in FIG. In this XYZ orthogonal coordinate system, the X direction and the Y direction are directions perpendicular to the optical axis 21a of the objective lens 21, respectively. Further, the Z direction is a direction parallel to the optical axis 21a of the objective lens 21. The optical axis 21a of the objective lens 21 coincides with the optical axis 4a of the illumination optical system 4. In each of the X direction, the Y direction, and the Z direction, the same side as the arrow is referred to as a + side (eg, +X side), and the opposite side is referred to as a − side (eg, −X side). In addition, when the optical path is bent by reflection, the directions corresponding to the X direction, the Y direction, and the Z direction are indicated with subscripts. For example, the Xa direction, the Ya direction, and the Za direction in FIG. 1 are directions corresponding to the X direction, the Y direction, and the Z direction in the optical path from the collimator lens 12 to the dichroic mirror 16, respectively.

The collimator lens 12 converts the excitation light L1 emitted from the optical fiber 11 into parallel light. The collimator lens 12 is arranged, for example, so that the focal point on the same side as the light source 3 coincides with the light emission port of the optical fiber 11. In the following description, regarding the lens included in the illumination optical system 4, the focus on the same side as the light source 3 is referred to as the rear focus, and the focus on the same side as the sample S is referred to as the front focus.

The λ / 4 wave plate 13 changes the polarization state of the excitation light L1 to circularly polarized light. The polarizer 14 is, for example, a polarizing plate, and has a characteristic of transmitting linearly polarized light having a predetermined polarization direction. The polarizer 14 is arranged such that the light incident on the sample S is S-polarized (linearly polarized in the Y direction). The polarizer 14 is rotatable around the optical axis 12a of the collimator lens 12. The optical axis 12a of the collimator lens 12 coincides with the optical axis 4a of the illumination optical system 4.

The mask 15 is a light beam splitting unit that splits the excitation light that excites the fluorescent substance into a plurality of light beams. The illumination optical system 4 scans the sample S with the interference fringe L2 formed by the interference of two or more light beams among the plurality of light beams divided by the mask 15. The mask 15 is arranged at or near the position of the pupil conjugate surface P1 optically conjugate with the pupil surface P0 of the objective lens 21. The vicinity of the pupil conjugate surface optically conjugate with the pupil surface P0 of the objective lens 21 is a range in which the excitation light L1 can be regarded as a parallel ray in the region including the pupil conjugate surface. For example, when the excitation light L1 is a Gaussian beam, it can be sufficiently regarded as a parallel light ray within a range within 1/10 of the Rayleigh length from the beam waist position. Rayleigh length of the wavelength of the excitation light L1 lambda, when the beam waist radius was w _0, is given by πw 0 ₂ ^/ λ. For example, when the wavelength of the excitation light L1 is 1 μm and the beam waist radius is 1 mm, the Rayleigh length is about 3 m, and the mask 15 is within 300 mm near the pupil conjugate plane P1 optically conjugate with the pupil plane P0 of the objective lens 21. It may be arranged. The mask 15 may be arranged at or near the pupil plane P0.

The mask 15 has an opening 15a and an opening 15b through which the excitation light L1 passes. Interference between the excitation light L1a passing through the opening 15a and the excitation light L1b passing through the opening 15b forms an interference fringe L2. The mask 15 is rotatable around the optical axis 12a of the collimator lens 12. The mask 15 is fixed, for example, relative to the polarizer 14 and rotates integrally with the polarizer 14. The mask 15 and the polarizer 14 are rotated by the torque supplied from the drive unit 22.

FIG. 2A is a diagram showing a mask according to the first embodiment. The openings 15a and 15b of the mask 15 are arranged symmetrically with respect to the optical axis 12a of the collimator lens 12 (see FIG. 1). In the state of FIG. 2A, the openings 15a and 15b are arranged in the Xa direction. FIG. 2B is a diagram showing the polarizer according to the first embodiment. The transmission axis 14a of the polarizer 14 is parallel to a direction (Ya direction in FIG. 2B) perpendicular to the direction (Xa direction in FIG. 2A) in which the openings 15a and 15b are aligned in the mask 15. Is set. FIG. 2C is a diagram showing the pupil plane P0 of the objective lens 21. Reference numerals P0a and P0b are regions where the excitation light L1 is incident, respectively. The parameters shown in FIG. 2C will be referred to later in the description of the image processing unit 7.

Returning to the description of FIG. 1, the dichroic mirror 16 has a property that the excitation light L1 is reflected and the fluorescence L3 (later shown in FIG. 4) from the sample S is transmitted. The excitation light L1 that has passed through the openings 15a and 15b of the mask 15 is reflected by the dichroic mirror 16, the optical path is bent, and enters the relay optical system 17. The relay optical system 17 guides the excitation light L1 from the dichroic mirror 16 to the scanning unit 18. Although the relay optical system 17 is represented by one lens in the figure, the number of lenses included in the relay optical system 17 is not limited to one. Further, the relay optical system 17 may not be necessary depending on the distance of the optical system and the like. In addition, in each drawing, two or more lenses may be represented by one lens even in a portion other than the relay optical system 17.

The scanning unit 18 scans the sample S with the interference fringes L2 formed by the excitation light L1 in the two directions of the X direction and the Y direction. The scanning unit 18 changes the position where the interference fringe L2 is formed by the excitation light L1 in two directions that intersect the optical axis 21a of the objective lens 21. The scanning unit 18 includes a deflection mirror 18a and a deflection mirror 18b. The tilts of the deflection mirror 18a and the deflection mirror 18b with respect to the optical path of the excitation light L1 are variable. The deflection mirror 18a and the deflection mirror 18b are a galvano mirror, a MEMS mirror, a resonant mirror (resonance type mirror), etc., respectively. The deflection mirror 18a and the deflection mirror 18b may be scanners. The scanning unit 18 is controlled by, for example, the control unit 8.

The deflection mirror 18a changes the position where the excitation light L1 is incident on the sample S in the X direction. The deflection mirror 18b changes the position on the sample S where the excitation light L1 is incident in the Y direction. The scanning unit 18 is arranged so that, for example, the position conjugate with the pupil plane P0 of the objective lens 21 is the position of the deflection mirror 18a, the position of the deflection mirror 18b, or the position between the deflection mirror 18a and the deflection mirror 18b. To be done. The scanning unit 18 may be configured such that the deflection mirror 18a changes the position where the excitation light L1 is incident on the sample S in the Y direction and the deflection mirror 18b changes in the X direction.

The excitation light L1 from the scanning unit 18 is incident on the lens 19. The lens 19 concentrates the excitation light L1 on the sample conjugate surface Sb optically conjugate with the sample surface Sa of the objective lens 21. The sample surface Sa is a surface that is arranged at a position near the front focal point or the front focal point of the objective lens 21 and is perpendicular to the optical axis 21a of the objective lens 21. An interference fringe is formed on the sample conjugate surface Sb due to the interference between the excitation light L1a passing through the opening 15a of the mask 15 and the excitation light L1b passing through the opening 15b.

The excitation light L1 that has passed through the sample conjugate surface Sb is incident on the lens 20. The lens 20 converts the excitation light L1 into parallel light. The excitation light L1 that has passed through the lens 20 passes through the pupil surface P0 of the objective lens 21. The objective lens 21 focuses the excitation light L1 on the sample surface Sa. The lens 20 and the objective lens 21 project the interference fringes formed on the sample conjugate surface Sb onto the sample surface Sa. Local interference fringes L2 are formed on the sample surface Sa.

The interference fringe L2 includes a bright portion having a relatively high light intensity and a dark portion having a relatively low light intensity. The direction in which the bright part and the dark part are lined up (X direction in FIG. 1) is appropriately referred to as the periodic direction D1 of the interference fringes L2. The periodic direction D1 of the interference fringe L2 corresponds to the direction in which the openings 15a and 15b of the mask 15 are arranged (Xa direction in FIG. 1). When the drive unit 22 rotates the mask 15 around the Za direction, the direction in which the openings 15a and 15b are aligned rotates, and the periodic direction D1 of the interference fringe L2 rotates around the Z direction. That is, the driving unit 22 is included in the fringe direction changing unit that changes the direction of the interference fringe L2. The drive unit 22 (stripe direction changing unit) is a direction in which two or more light beams are arranged in a plane perpendicular to the optical axis 4a of the illumination optical system 4 (for example, the surface on the light emission side of the mask 15) (hereinafter referred to as a light beam division direction). ) Is changed. The above-described light beam splitting direction is, for example, a direction in which the opening 15a and the opening 15b are arranged, and the drive unit 22 changes the light beam splitting direction by rotating the mask 15.

Further, when the mask 15 rotates around the Za direction, the direction in which the excitation light L1 enters the sample S changes. The drive unit 22 rotates the polarizer 14 in conjunction with the mask 15 to change the direction of the transmission axis of the polarizer 14 and adjusts the excitation light L1 to be S-polarized and incident on the sample S. That is, the polarizer 14 and the driving unit 22 are included in the polarization adjusting unit that adjusts the polarization state of the excitation light L1 based on the direction of the interference fringes.

FIG. 3 is a diagram showing the mask, the polarizer, the interference fringes, and the polarization state of the excitation light according to the first embodiment. In FIG. 3A, the direction in which the openings 15a and 15b of the mask 15 are arranged is the Xa direction. The transmission axis 14a of the polarizer 14 is in the Ya direction perpendicular to the Xa direction. In this case, in the excitation light L1 (see FIG. 1), the light flux passing through the opening 15a and the light flux passing through the opening 15b are incident on the sample S to generate the interference fringes L2 in the periodic direction D1. The excitation light L1 incident plane is parallel to the XZ plane. The excitation light L1 when incident on the sample S is in the Y direction whose polarization direction D2 is perpendicular to the incident surface, that is, the excitation light L1 is incident on the sample S with S polarization.

In FIG. 3B, the direction in which the openings 15a and 15b of the mask 15 are aligned is the direction obtained by rotating the Xa direction counterclockwise by 120°. The transmission axis 14a of the polarizer 14 is a direction in which the Ya direction is rotated counterclockwise by 120 °. The periodic direction of the interference fringes L2 is a direction forming 120 ° with respect to the X direction. The incident surface of the excitation light L1 is a surface obtained by rotating the XZ plane by 120 ° around the Z direction. The excitation light L1 when entering the sample S has a polarization direction D2 perpendicular to the incident surface, that is, the excitation light L1 enters the sample S as S-polarized light.

In FIG. 3C, the direction in which the openings 15a and 15b of the mask 15 are aligned is the direction obtained by rotating the Xa direction counterclockwise by 240°. The transmission axis 14a of the polarizer 14 is a direction in which the Ya direction is rotated counterclockwise by 240 °. The periodic direction D1 of the interference fringe L2 is a direction forming 240 ° with respect to the X direction. The incident surface of the excitation light L1 is a surface obtained by rotating the XZ plane by 240 ° around the Z direction. The excitation light L1 when entering the sample S has a polarization direction D2 perpendicular to the incident surface, that is, the excitation light L1 enters the sample S as S-polarized light.

As described above, when the excitation light L1 is incident on the sample S with S-polarized light, the contrast of the interference fringes L2 is higher than when it is incident with P-polarized light. In FIG. 3, the periodic direction of the interference fringe L2 is changed in three ways in increments of 120 °, but the periodic direction of the interference fringe L2 is not limited to this example. The periodic direction of the interference fringes L2 corresponds to a direction in which the resolution can be improved (a direction in which the super-resolution effect can be obtained) in the image generated by the image processing unit 7 described later. The periodic direction of the interference fringe L2 is appropriately set so that a desired super-resolution effect can be obtained. For example, the periodic directions of the interference fringes L2 may be two or one at an angle of 90 ° to each other. Further, the mask 15 may be replaceable according to the magnification and NA (numerical aperture) of the objective lens 21 and the shape of the illumination pupil.

FIG. 4 is a diagram showing the microscope and the optical path of fluorescence according to the first embodiment. The detection optical system 5 forms an image of the fluorescence L3 generated in the sample S. The detection optical system 5 includes an objective lens 21, a lens 20, a lens 19, a scanning unit 18, a relay optical system 17, a dichroic mirror 16, an excitation light cut filter 23, and a lens 24 in order from the sample S toward the detection device 6. .. The fluorescence L3 generated in the sample S passes through the objective lens 21, the lens 20, and the lens 19 in this order and enters the scanning unit 18. The fluorescence L3 is descanned by the scanning unit 18 and enters the dichroic mirror 16 through the relay optical system 17. The dichroic mirror 16 has the property of transmitting the fluorescence L3. The fluorescence L3 transmitted through the dichroic mirror 16 is incident on the excitation light cut filter 23. The excitation light cut filter 23 has a property of blocking the excitation light L1 and transmitting the fluorescence L3. The fluorescence L3 that has passed through the excitation light cut filter 23 enters the lens 24. The lens 24 focuses the fluorescence L3 on the detection device 6.

The detection device 6 is an image sensor and includes a plurality of detection units 6a arranged two-dimensionally. The plurality of detection units 6a are arranged in two directions in the detection device 6. The plurality of detection units 6a are arranged in two directions, the Xb direction and the Yb direction. Each of the plurality of detection units 6a is a sensor cell including a photoelectric conversion element such as a photodiode, a pixel, or a photodetector. Each of the plurality of detection units 6a can detect the fluorescence L3. The detection unit 6a corresponds to, for example, one pixel, but a detection region (light receiving region) including a plurality of pixels may be used as one detection unit 6a.

The microscope 1 scans the interference fringes L2 on the sample surface Sa by the scanning unit 18, and the detection device 6 detects the fluorescence L3. For example, the microscope 1 illuminates the illumination region selected from the sample surface Sa with the interference fringes L2, and the detection device 6 detects the fluorescence L3 from the illumination region. The microscope changes the illumination area by the scanning unit 18 after the detection by the detection device 6 is completed. The microscope 1 acquires a fluorescence intensity distribution (measured value of the detection device 6) in a desired region by repeating a process of detecting fluorescence and a process of changing the illumination region.

In the embodiment, the fluorescent substance contained in the sample S is multiphoton excited by the excitation light L1. In the case of multiphoton excitation, the wavelength band of the excitation light L1 can be extended to the wavelength band of the infrared light, so that the transmittance of the excitation light L1 to the sample S becomes high. Further, since the fluorescence L3 to be detected is generated only in the vicinity of the focal plane of the objective lens 21, it is possible to accurately obtain image data on the sample surface Sa set in an arbitrary portion including the inside of the sample S. it can.

The image processing unit 7 generates an image based on the detection result of the detection device 6 obtained as described above. Hereinafter, the processing executed by the image processing unit 7 will be described. In the mathematical formulas used in the following description, the coordinate system is described as a vector as appropriate. The coordinates on the sample surface Sa and the coordinates on the detection device 6 (hereinafter referred to as detector coordinates) are represented by a vector r=(x, y), and the corresponding wave number coordinates (coordinates after Fourier transform at r) are represented by a vector k=(k). _x, represented by _{k y).} Further, coordinates of the scanning destination of the scanning unit 18 (hereinafter, referred to as scanning coordinates) are represented by a vector r _s =(x _s , y _s ) and corresponding wave number coordinates (coordinates after Fourier transform at r _s ) are vectors. It is represented by k _s =(k _xs , k _ys ). In the following description, the wave number may be referred to as spatial frequency or frequency. The magnification of the optical system is assumed to be 1 for convenience of explanation, but any magnification may be used.

When the numerical aperture of the optical system including the objective lens 21 is NA, the illumination light wavelength is λ _ex , and the wavelength of the fluorescence L3 is λ _em , the pupil radius k _NA ^ex of the objective lens 21 when the excitation light is incident and the fluorescence is incident. In such a case, the pupil radius k _NA ^em of the objective lens 21 is expressed by the following formula (1A) and formula (1B). As is well known, since the electric field amplitudes of the pupil plane and the image plane are connected by the Fourier transform relationship, the coordinates of the pupil position may be represented by wave number coordinates. Each of k _NA ^ex and k _NA ^em indicates the value of the pupil radius when the pupil is expressed in wave number coordinates.

Here, various parameters will be described with reference to FIG. 2C. In FIG. 2C, the pupil plane P0 is represented by a wave number coordinate space (frequency space). The area inside the circle drawn by the dotted line shown in FIG. 2C is the pupil of the objective lens 21, and k _NA ^ex is the pupil radius of the objective lens 21. The region P0a and the region P0b on which the excitation light L1 is incident are assumed to be circular, but are not limited to circular. The radius of each of the region P0a and the region P0b is σk _NA ^ex . σ is the ratio of the radius of the region P0a or the region P0b to the pupil radius of the objective lens 21. The distance from the optical axis 21a of the objective lens 21 to the center of the region P0a is (1-σ) k _NA ^ex . The distance between the center of the region P0a and the center of the region P0b is, for example, 2(1-σ)k _NA ^ex , but is not limited to this value. The electric field intensity ill(r) of the excitation light on the sample surface Sa is expressed by the following equation (2).

Here, the vector k ₀ =(k ₀ , 0) represents the wave number vector of the illumination fringe, and k ₀ =2(1-σ)k _NA ^ex . PSF _ill (r) is a point spread function when the numerical aperture of the optical system is σNA. The distance between the interference fringes of ill (r) (distance from one bright part to the next bright part) is 1 / k ₀ = 1 / (2 (1-σ) k _NA ^ex ). In the following description, the interval of the interference fringes will be appropriately referred to as the fringe interval or the period of the interference fringes.

In the embodiment, the fluorescent substance contained in the sample S is multiphoton excited by the excitation light L1, and the excited fluorescent substance emits the fluorescence L3. The detection device 6 receives the fluorescence L3 and captures an image of the fluorescent substance formed by the detection optical system 5. The detection device 6 captures an image of a fluorescent substance and acquires image data. In the following description, the size of the detection unit 6a of the detection device 6 (detection unit size) is smaller than the size corresponding to the cycle of the interference fringes L2 in the detection device 6 (the length on the detection device 6 corresponding to one cycle). Is small enough. For example, the size of the detection unit 6a is set to about λ _em /4NA.

Here, the distribution of the fluorescent substance in the sample S is represented by Obj(r), and the image data obtained by the detection device 6 when detecting the fluorescence generated by n-photon excitation is represented by I(r,r _s ). I(r,r _s ) is represented by the following equation (3).

* ^R in equation (3) is a convolution for r. Here, the PSF _det (r) is a detection PSF determined by the detection optical system 5 including the objective lens 21 and the fluorescence wavelength λ _em . Further, the term of the excitation light intensity ill (r) to the nth power appears in the equation (3), reflecting that the fluorescence intensity generated by the n photon excitation is proportional to the nth power of the excitation light intensity. Image data I _(r, r _s) is a detector coordinate r = (x, y) and the scan coordinates _{r s = (x s, y} s) 4 -dimensional data with the independent variables. When I(r,r _s ) is transformed, the following equation (4) is obtained.

^{* Rs} in the equation (4) is a convolution of _{r s. Further, PSF} eff (r, _{r s)} is the effective PSF which is defined by the following equation (5).

From the above formula (4), it can be seen that different image data of Obj(r _s ) can be obtained for each detection unit 6a of the detection device 6. Further, from the above formula (5), it can be seen that the shape of the effective PSF is different for each position (r) of the detection unit 6a of the detection device 6.

FIG. 5 is a diagram showing effective PSF in each detection unit of the detection device according to the first embodiment. In each graph of FIG. 5, the horizontal axis is the Xb direction of the detection device 6. The sample surface Sa is optically conjugate with the detection device 6, and the coordinates X of the sample surface Sa and the coordinates Xb of the detection device are associated with each other by an appropriate coordinate conversion. For example, when the magnification of the optical system is 1, X=Xb.

In FIG. 5A, the effective PSF (solid line) of each detection unit 6a is shown in one graph for the three detection units 6a whose coordinates in the Xb direction are different from each other. For example, the graph in the center of FIG. 5A shows the distribution Q1a (solid line) corresponding to the effective PSF of the detection unit 6a arranged at the position X1a. The graph on the left side of FIG. 5A shows the distribution Q1b (solid line) corresponding to the effective PSF of the detection unit 6a arranged at the position X1b. The graph on the right side of FIG. 5A shows the distribution Q1c (solid line) corresponding to the effective PSF of the detection unit 6a arranged at the position X1c.

Reference numeral Q2 corresponding to the dotted line in FIG. 5 is a distribution corresponding to the square of the intensity distribution of the interference fringe L2 shown in FIG. The distribution Q2 corresponds to the electric field intensity ill ² (r) of the excitation light on the sample surface Sa (the square of the above equation (3)). In FIG. 5, the case of two-photon excitation is shown as an example, but in the case of n-photon excitation, Q2 has a distribution corresponding to the n-th power of the electric field intensity ill (r) of the interference fringe L2. The position where the intensity of the interference fringe L2 is maximum, that is, the peak positions X2a, X2b, and X2c of the distribution Q2 can be obtained in advance by numerical simulation or the like.

Distribution Q2 includes partial distributions Q2a, Q2b, and Q2c. The distribution Q2a is a distribution in the range from the minimum position before the peak position X2a to the next minimum position. The distribution Q2b is a distribution in the range from the minimum position before the peak position X2b to the next minimum position. The distribution Q2c is a distribution in the range from the minimum position before the peak position X2c to the next minimum position.

Reference numerals Q3a, Q3b, and Q3c corresponding to the chain double-dashed line in FIG. 5 are distributions corresponding to the detection optical system 5 including the objective lens 21 and the detection PSF determined by the fluorescence wavelength λ _em . The detected PSF corresponds to PSF _det (r) such as in equation (3).

The distribution Q3a shown in the center graph of FIG. 5A is a distribution corresponding to the detection PSF of the detection unit 6a arranged at the position X1a among the plurality of detection units 6a. The distribution Q3a has a maximum (peak) at the position X1a where the detector 6a is arranged (eg, the center position of the light receiving region of the detector 6a). The position X1a is almost the same as the peak position X2a of the distribution Q2a corresponding to the intensity distribution of the interference fringe L2. The distribution Q1a corresponding to the effective PSF is a distribution obtained by multiplying the distribution Q2 corresponding to the intensity distribution of the interference fringes L2 and the distribution Q3a corresponding to the detected PSF of the detection unit 6a arranged at the position X1a.

In the graph in the center of FIG. 5A, the position X1a of the detection unit 6a, that is, the peak position of the detection PSF (the peak position of the distribution Q3a) is deviated from the peak position X2a of the distribution Q2a corresponding to the intensity distribution of the interference fringes L2. The amount is smaller than the predetermined value (for example, the shift amount is almost 0). In such a case, the effective PSF distribution Q1a has a single maximum (peak). In this case, the peak position of the distribution Q1a is almost the same as the position X1a of the detection unit 6a or the peak position X2a of the distribution Q2a corresponding to the intensity distribution of the interference fringes L2.

The distribution Q3b shown in the graph on the left side of FIG. 5A is a distribution corresponding to the detection PSF of the detection unit 6a arranged at the position X1b among the plurality of detection units 6a. The distribution Q3b reaches a maximum (peak) at the position X1b where the detection unit 6a is arranged (for example, the center position of the light receiving region of the detection unit 6a). The position X1b is displaced from the peak position X2b of the partial distribution Q2b including the position X1b in the distribution Q2 corresponding to the intensity distribution of the interference fringe L2. The distribution Q1b corresponding to the effective PSF is a distribution obtained by multiplying the distribution Q2 corresponding to the intensity distribution of the interference fringe L2 and the distribution Q3b corresponding to the detected PSF of the detection unit 6a arranged at the position X1b.

In the graph on the left side of FIG. 5A, the position X1b of the detection unit 6a, that is, the peak position of the detected PSF (peak position of the distribution Q3b) is deviated from the peak position X2b of the distribution Q2b corresponding to the intensity distribution of the interference fringe L2. The amount is larger than the predetermined value. In this case, the distribution Q1b of the effective PSF has two maxima (peaks). As described above, the peak of the effective PSF may be divided into two depending on the position of the detector 6a, and such a change in the shape of the effective PSF is called collapse of the shape of the effective PSF. The strongest peak of the effective PSF is called the main lobe, and the other peaks are called the side lobes.

The peak position of the main lobe of the effective PSF distribution Q1b deviates from the center position (X2a) of the detection device 6. As described above, it can be seen that the position of the main lobe of the effective PSF also shifts due to the relationship between the position (r) of the detection unit 6a of the detection device 6 and the position of the intensity distribution of the interference fringe L2. In the following description, the displacement of the main lobe of the effective PSF is appropriately referred to as the displacement of the effective PSF.

The distribution Q3c shown in the graph on the right side of FIG. 5A is a distribution corresponding to the detection PSF of the detection unit 6a arranged at the position X1c among the plurality of detection units 6a. The distribution Q3c has a maximum (peak) at the position X1c where the detector 6a is arranged (eg, the center position of the light receiving region of the detector 6a). The position X1c deviates from the peak position X2c of the partial distribution Q2c including the position X1c in the distribution Q2 corresponding to the intensity distribution of the interference fringes L2. The distribution Q1c corresponding to the effective PSF is a distribution obtained by multiplying the distribution Q2 corresponding to the intensity distribution of the interference fringes L2 and the distribution Q3c corresponding to the detected PSF of the detection unit 6a arranged at the position X1c.

In the graph on the right side of FIG. 5A, the position X1c of the detection unit 6a, that is, the peak position of the detection PSF (the peak position of the distribution Q3c) is deviated from the peak position X2c of the distribution Q2c corresponding to the intensity distribution of the interference fringes L2. The amount is larger than the predetermined value. In this case, the distribution Q1b of the effective PSF has two maximums (peaks), the shape of the effective PSF is deformed, and the position of the effective PSF is displaced.

In FIG. 5(B), the position of the detection unit 6a is different from that in FIG. 5(A). The distribution Q3d shown in the graph on the left side of FIG. 5B is a distribution corresponding to the detection unit 6a arranged at the position X1d among the plurality of detection units 6a. The distribution Q3d has a maximum (peak) at the position X1d where the detector 6a is arranged (eg, the center position of the light receiving region of the detector 6a). The position X1d is substantially the same as the peak position X2b of the partial distribution Q2b including the position X1d in the distribution Q2 corresponding to the intensity distribution of the interference fringe L2. In such a case, the distribution Q1d of the effective PSF has a single maximum (peak), and the peak position of the distribution Q1d is the peak position of the distribution Q2b corresponding to the position X1d of the detection unit 6a or the intensity distribution of the interference fringe L2. It becomes almost the same as X2b. That is, the shape of the effective PSF has not collapsed.

Further, the distribution Q3e shown in the graph on the right side of FIG. 5B is a distribution corresponding to the detection unit 6a arranged at the position X1e among the plurality of detection units 6a. The distribution Q3e has a maximum (peak) at the position X1e (for example, the center position of the light receiving region of the detection unit 6a) where the detection unit 6a is arranged. The position X1e is substantially the same as the peak position X2c of the partial distribution Q2c including the position X1e in the distribution Q2 corresponding to the intensity distribution of the interference fringes L2. In such a case, the distribution Q1e of the effective PSF has a single maximum (peak), and the peak position of the distribution Q1e is the peak position of the distribution Q2c corresponding to the position X1e of the detection unit 6a or the intensity distribution of the interference fringe L2. It becomes almost the same as X2c. That is, the shape of the effective PSF has not collapsed.

In the present embodiment, the image processing unit 7 uses the detection result of the detection unit 6a selected from the plurality of detection units 6a based on the magnification of the detection optical system 5 and the period (strip interval) of the interference fringes L2. The image processing unit 7 selects the detection unit 6a from the plurality of detection units 6a based on the peak position of the interference fringe L2 (eg, peak positions X2a, X2b, X2c in FIG. 5), and the detection result of the selected detection unit 6a. To use. The peak position of the interference fringe L2 corresponds to, for example, the position where the intensity is maximum in the intensity distribution of the interference fringe L2 (eg, the central position of the bright portion).

The image processing unit 7 uses, for example, the detection result of the detection unit 6a arranged at the position X1a corresponding to the peak position X2a as the detection result corresponding to the peak position X2a in the center graph of FIG. For example, the peak positions X2a, X2b, and X2c are obtained in advance by numerical simulation or the like, and are stored in the storage unit in advance. The image processing unit 7 selects the detection unit 6a arranged closest to the peak position X2a among the plurality of detection units 6a based on the stored information on the peak position, and displays the detection result of the selected detection unit 6a. To use.

The image processing unit 7 may use only the detection result of the one detection unit 6a arranged at the position X1a as the detection result regarding the partial distribution Q2a including one peak in the intensity distribution of the interference fringe L2. The detection result of the detection unit 6a arranged at the position X1a and at least one detection unit 6a around the detection unit 6a may be used.

The image processing unit 7 uses, for example, the detection result of the detection unit 6a arranged at the position X1d corresponding to the peak position X2b as the detection result corresponding to the peak position X2b in the graph on the left side of FIG. 5B. .. Based on the magnification of the detection optical system 5 and the cycle of the interference fringes L2, the image processing unit 7 includes a plurality of detectors 6a whose positions match the partial distribution Q2b including one peak in the intensity distribution of the interference fringes L2. Select from the detection unit 6a. For example, the image processing unit 7 may detect the detection unit 6a arranged closest to the peak position X2b (for example, the detection unit arranged at the position X1d) among the plurality of detection units 6a based on the stored information about the peak position. 6a) is selected. The image processing unit 7 uses the detection result of the selected detection unit 6a as the detection result regarding the distribution Q2b.

The image processing unit 7 may use only the detection result of the one detection unit 6a arranged at the position X1d as the detection result regarding the partial distribution Q2b including one peak in the intensity distribution of the interference fringe L2. The detection results of the detection unit 6a arranged at the position X1d and at least one detection unit 6a around the detection unit 6a may be used. In the effective PSF (PSF _eff ) corresponding to the distribution Q1d, the collapse of the shape of the effective PSF is reduced by matching the peak position X2b of the distribution Q2b with the position X1d of the detection unit 6a.

Further, the image processing unit 7 uses, for example, the detection result of the detection unit 6a arranged at the position X1e corresponding to the peak position X2c as the detection result corresponding to the peak position X2c in the graph on the right side of FIG. 5B. .. Based on the magnification of the detection optical system 5 and the period of the interference fringes L2, the image processing unit 7 includes a plurality of detectors 6a whose positions match the partial distribution Q2c including one peak in the intensity distribution of the interference fringes L2. Select from the detection unit 6a. For example, the image processing unit 7 may detect the detection unit 6a (eg, the detection unit arranged at the position X1e) closest to the peak position X2c among the plurality of detection units 6a based on the stored peak position information. 6a) is selected. The image processing unit 7 uses the detection result of the selected detection unit 6a as the detection result regarding the distribution Q2c.

The image processing unit 7 may use only the detection result of the one detection unit 6a arranged at the position X1e as the detection result regarding the partial distribution Q2c including one peak in the intensity distribution of the interference fringe L2, You may use the detection result of the detection part 6a arrange|positioned at the position X1e, and at least one detection part 6a around this detection part 6a. In the distribution Q1e(PSF _eff ), the peak position X2c of the distribution Q2c and the position X1e of the detection unit 6a are matched with each other, whereby the collapse of the shape of the effective PSF is reduced.

The image processing unit 7 corrects the image position deviation (effective PSF _eff peak position or main lobe position deviation) for each detection unit with respect to the detection result of the detection unit 6a selected as described above. The positional deviation of the image for each detection unit can be acquired by theoretical calculation using various design values or from a captured image obtained by photographing a small object such as fluorescent beads with the detection device 6. By correcting such misalignment, the effective PSF of the images obtained by each of the selected detection units 6a can be made substantially the same. The PSF _eff of the image obtained in this way can be approximately regarded as equivalent to the PSF _eff of the detection unit (detection unit located on the optical axis) at the center position of the detection device 6. The PSF _eff of the detection unit at the central position (r=(0,0)) of the detection device 6 is represented by the following equation (6) with reference to the equation (5).

Focusing on the period direction of the interference fringes L2, that is, the k ₀ direction, it can be seen from Equation (6) that the smaller the period of the interference fringes L2, the narrower the full width at half maximum of PSF _{eff and} the better the resolution. That is, as the number of fringes (bright part) included in the periodic direction of the interference fringe L2 in the embodiment is larger, the full width at half maximum of PSF _eff is narrower and the resolution is better. From the viewpoint of resolution, the number of fringes (bright areas) included in the periodic direction of the interference fringes L2 is preferably 3 or more, for example.

The image processing unit 7 generates an image by adding the images having the same PSF _eff . The image processing unit 7 can generate the image I _SR (r _s ) having good resolution and S/N ratio because the PSF _{eff of the} images to be added are almost the same. Incidentally, when a wide range of the detection unit 6a for use in the generation of the image I _{SR (r s),} it is possible to increase the signal amount. Moreover, when narrowing the range of the detection unit 6a for use in the generation of the image I _{SR (r s),} it is possible to increase the sectioning capabilities.

The effective OTF can be obtained by Fourier-transforming the above equation (6). The _cut- off frequency k _cut ^conv in an ordinary fluorescence microscope is given by k _cut ^conv =2NA/λ. The microscope according to the embodiment has a cutoff frequency of up to 2k _cut ^conv with the OTF magnified in the direction of the interference fringes L2. Here, for the sake of simplicity, it is assumed that the fluorescence wavelength is λ and the excitation wavelength is n × λ in the case of n photon excitation. The OTF according to the embodiment is a combination of the OTF of the ordinary fluorescence microscope and the component of the OTF of the ordinary fluorescence microscope shifted in the periodic direction of the interference fringes L2.

As described in the above equation (6), the microscope 1 according to the embodiment improves the resolution of the interference fringes L2 in the periodic direction (X direction in FIG. 1). The microscope 1 can also improve the resolution two-dimensionally by detecting the fluorescence from the sample S by changing the periodic direction of the interference fringes L2. Here, an example in which the cycle direction of the interference fringe L2 is changed by 90° will be described. To change the period direction of the interference fringe L2 by 90°, the mask 15 and the polarizer 14 in the state of FIG. 2 are rotated by 90° around the Za direction.

The super-resolution image when the periodic direction of the interference fringes L2 is the X direction and I _{SRx (r s),} the super-resolution image when the periodic direction of the interference fringes L2 is the Y direction I _{SRy (r} s) And The image processing unit 7, by summing the I _{SRx (r s)} and I _{SRy (r s),} may generate a super-resolution image two-dimensionally resolution is improved. The image processing unit 7 may also generate a super-resolution image by the following processing.

The image processing section 7 performs Fourier transform super-resolution image I _{SRx (r s)} and a super-resolution image I _SRy the _{(r s),} respectively. Here, the Fourier-transformed super-resolution image I _SRx (r _s ) is represented by I ^to _SRx (k _s ). " ^~ " In the specification is a tilder in the mathematical formula. In addition, the Fourier-transformed super-resolution image I _SRy (r _s ) is represented by I 1 ^to _SRy (k _s ). I ^~ _SRx (k _s), as compared to conventional fluorescence microscope, the cutoff frequency is increased with respect to the periodic direction of the interference fringes (X direction). Also, ^{I ~} _{SRy (k s),} as compared to conventional fluorescence microscope, the cutoff frequency is increased with respect to the periodic direction of the interference fringes (Y-direction). The image processing section 7 adds up the I ~ _{SRx (k s)} and I ~ _{SRy (k s).} This increases the cutoff frequency in the two directions (X direction and Y direction).

The added effective OTF shape may be distorted depending on the combination of the directions that change the periodic direction of the interference fringes L2. In this case, the image processing unit 7 may apply a frequency filter for correcting the shape of the effective OTF. Thus, it is possible to improve the effective resolution than when summing the I _{SRx (r s)} and I _{SRy (r s).} Further, as described with reference to FIG. 3, the illumination optical system 4 changes the periodic direction of the interference fringe L2 into three types of 0°, 120°, and 240°, and the detection device 6 has three periodic directions. May detect the fluorescence L3. The image processing unit 7 may generate a super-resolution image by using three detection results (for example, three images) detected by the detection device 6 in three different cycle directions. Further, the illumination optical system 4 changes the periodic direction of the interference fringe L2 in four or more ways, the detection device 6 detects the fluorescence L3 in each of the four or more periodic directions, and the image processing unit 7 has four ways. A super-resolution image may be generated by using four or more detection results detected by the detection device 6 in the above periodic directions. The method of generating a super-resolution image using a plurality of detection results obtained by changing the cycle direction of the interference fringes a plurality of times described above can also be used in the following embodiments.

Next, the observation method according to the embodiment will be described based on the configuration of the microscope 1 described above. FIG. 6 is a flowchart showing the observation method according to the embodiment. For each part of the microscope 1, refer to FIG. 1 or FIG. 4 as appropriate. In step S1, the control unit 8 of the microscope 1 sets the angles of the scanning mirrors (deflection mirrors 18a and 18b). The illumination optical system 4 of FIG. 1 irradiates the position on the sample determined by the angle of the scanning mirror set in step S1 with the excitation light as interference fringes. In step S2, the fluorescent substance of the sample is excited by the interference fringes of the excitation light. In step S3, the detection device 6 of FIG. 4 detects the fluorescence L3 from the sample S via the detection optical system 5.

In step S4, the control unit 8 determines whether or not to change the angle of the scanning mirror. When it is determined that the processing of steps S1 to S3 has not been completed for a part of the scheduled observation region, the control unit 8 determines to execute the angle change of the scanning mirror in step S4 (step S4; Yes). ). When the control unit 8 determines to change the angle of the scanning mirror (step S4; Yes), the process returns to step S1 and the control unit 8 sets the angle of the scanning mirror to the next scheduled angle. Then, the processes of steps S2 to S4 are repeated. In this way, the illumination optical system 4 scans the sample S two-dimensionally with the interference fringes of the excitation light L1.

When it is determined in step S4 that the processing of steps S1 to S3 has been completed for all of the scheduled observation regions, the control unit 8 determines not to change the angle of the scanning mirror (step S4; No). .. When the control unit 8 determines not to change the angle of the scanning mirror (step S4; No), the image processing unit 7 corrects the positional deviation of the image for each detection unit in step S5. The image processing unit 7 corrects the data obtained from at least one detection unit of the plurality of detection units based on the position of the detection unit. For example, the image processing unit 7 corrects the data obtained from the detection unit selected from the plurality of detection units based on the position of the detection unit. For example, the image processing unit 7 uses the data obtained from the first detection unit (for example, the detection unit arranged at the position X1d in FIG. 5B) of the plurality of detection units as the first detection unit. Correction is performed based on the position (eg, X1d). The image processing unit 7 also generates an image using the detection results of two or more detection units. For example, in step S6, the image processing unit 7 generates an image (for example, a super-resolution image) by adding the images corrected in step S5.

The positions of the plurality of detection units 6a of the detection device 6 may be set based on the cycle of the interference fringe L2 so as to match the peak (or maximum or bright point) position of the interference fringe L2. The detection device 6 may be preset so that the spacing between the detection units 6a and the fringe spacing of the interference fringes L2 match. The distance between the detection units 6a is the distance between the center of one detection unit 6a and the center of the detection unit 6a adjacent to the center. The fringe spacing of the interference fringe L2 is the spacing between the centerline of one bright portion and the centerline of the adjacent bright portion in the interference fringe L2. Here, assuming that the wave number of the interference fringe L2 is k ₀ , the fringe interval of the interference fringe L2 is 1 / k ₀ . When the fringe spacing of the interference fringes is 1/k ₀ , the spacing of the detection unit 6a of the detection device 6 is set to be substantially the same as P shown in the following formula (7).

In the above formula (7), the magnification of the detection optical system 5 including the objective lens 21 is set to 1. If the magnification of the detection optical system 5 is _{M det} is changing the spacing between the detectors 6a by a factor fraction, the spacing of the detectors 6a may be set to _M det / _{k 0.} Alternatively, by using a part of the detection optical system 5 as a zoom lens, the distance between the detectors 6a can be matched with the period of the interference fringes L2. In this case, it is desirable that the lens 24 capable of changing only the magnification of the detection optical system 5 is a zoom lens. Further, the period of the interference fringes L2 may be adjusted so as to match the intervals of the plurality of detectors 6a of the detection device 6. For example, the period of the interference fringes L2 can be changed by changing the intervals between the openings 15a and 15b of the mask 15.

According to the first embodiment, the effective PSFs of the images obtained for each detection unit 6a are aligned by setting the intervals of the detection units 6a and the fringe intervals of the interference fringes L2 to match. Therefore, the image processing unit 7 generates images with high resolution (eg, super-resolution images) while ensuring S/N by adding images based on the detection results of two or more detection units. You can Further, the fluorescent substance contained in the sample S is multiphoton excited by the excitation light L1. In the case of multiphoton excitation, the wavelength band of the excitation light L1 can be extended to the wavelength band of the infrared light, so that the transmittance of the excitation light L1 to the sample S becomes high. Further, since the fluorescence L3 to be detected is generated only in the vicinity of the focal plane of the objective lens 21, it is possible to accurately obtain image data on the sample surface Sa set in an arbitrary portion including the inside of the sample S. it can.

In the present embodiment, the microscope 1 scans the interference fringes L2 two-dimensionally by scanning the interference fringes L2 in two directions parallel to the sample surface Sa. The microscope 1 according to the embodiment may scan the interference fringes L2 three-dimensionally by scanning the interference fringes L2 in two directions parallel to the sample surface Sa and one direction perpendicular to the sample surface Sa. When the interference fringes L2 are three-dimensionally scanned, the process of scanning the interference fringes L2 in two directions parallel to the sample surface Sa (hereinafter referred to as the two-dimensional process) is the same as the process described in the above embodiment. is there. The microscope 1 can generate, for example, a three-dimensional super-resolution image by changing the position in the Z direction and repeating the two-dimensional processing. Similarly for the embodiments described later, the microscope 1 may scan the interference fringes L2 three-dimensionally.

[Second Embodiment]
The second embodiment will be described. In the present embodiment, the same components as those in the above-described embodiment will be designated by the same reference numerals, and the description thereof will be omitted or simplified. In the present embodiment, the image processing unit 7 (see FIG. 4) filters the data in the frequency space to generate an image. The image processing unit 7 performs deconvolution on the data obtained from the detection device 6 to generate an image. The image processing unit 7 performs deconvolution and apodization for each detection unit 6a of the detection device 6 as the above-described filtering to generate an image. That is, the image processing unit 7 performs filtering including deconvolution on the data in the frequency space. In the following description, a series of processes of deconvolution and apodization may be collectively (generally) referred to as deconvolution.

FIG. 7 is a diagram showing processing of the image processing unit of the microscope according to the second embodiment. For each part of the microscope, refer to FIG. 1 or FIG. 4 as appropriate. FIG. 7 (A) is the PSF before deconvolution, which is the same as FIG. 5 (A). In FIG. 7A, the distance between the detection units 6a of the detection device 6 (see FIG. 4) does not match the distance between the interference fringes L2. In this case, the shape of the effective PSF (solid line) of the image obtained for each detection unit 6a is deformed depending on the position of the detector 6a, as described in the above equation (5). The effective PSF for each detector 6a can be obtained (estimated) by theoretical calculation from the design value or by photographing a small object such as fluorescent beads. The image processing unit 7 uses the thus obtained effective PSF to perform deconvolution so as to correct the shape collapse and the positional deviation of the effective PSF of the image obtained for each detection unit 6a.

Figure 7 (B) shows the PSF after deconvolution. In the graph in the center of FIG. 7B, reference sign Q4a is an effect obtained by deconvoluting the effective PSF of the detection unit 6a arranged at the distribution Q1a shown in the graph in the center of FIG. 7A, that is, the position X1a. Corresponds to PSF. Here, the amount of deviation between the position X1a of the detection unit 6a and the peak position X2a of the distribution Q2a is smaller than a predetermined value, and the distribution Q4a corresponding to the effective PSF after deconvolution corresponds to the effective PSF before deconvolution. It is almost the same as the distribution Q1a.

Further, in the graph on the left side of FIG. 7B, reference sign Q4b is obtained by deconvoluting the distribution Q1b shown in the graph on the left side of FIG. 7A, that is, the effective PSF of the detection unit 6a arranged at the position X1b. Corresponds to the effective PSF that is In the graph on the right side of FIG. 7(B), reference sign Q4c indicates the distribution Q1c shown in the graph on the right side of FIG. 7(A), that is, the effective PSF of the detection unit 6a arranged at the position X1c is obtained by deconvolution. Corresponds to PSF. By such a series of processes (deconvolution), the effective PSFs of the detectors 6a become substantially the same, as shown in the three graphs of FIG. 7B. The image processing unit 7 uses the result of deconvolution to generate an image. Hereinafter, the processing of the image processing unit 7 will be described in more detail.

The image processing unit 7 converts at least a part of the detection results of the plurality of detection units 6a into data on the frequency space, and uses the conversion results to generate an image (eg, a super-resolution image). In the following description, data representing at least a part of the detection results of the plurality of detection units 6a in the frequency space is appropriately referred to as a component of the frequency space. The image processing unit 7 Fourier transforms at least a part of the detection results of the plurality of detection units 6a, and generates an image using the components of the frequency space obtained by the Fourier transform. When the above equation (5) to Fourier transform for r _s, the following equation (8) is obtained.

I 1 ^to (r, k _s ) on the left side of the equation (8) is the Fourier transform of I (r, r _s ) with respect to r _s . Right side of the _OTF eff (r, _{k s) is, PSF eff (r, r s} ) the is obtained by Fourier transform for _{r s,} represents the effective OTF of each detector 6a of the detector 6. In addition, the right-hand side of ^Obj ~ _{(k s)} are those Obj a _{(r s)} obtained by Fourier transform on _{r s.}

There are various deconvolution methods such as the Wiener filter and Richardson-Lucy method. Here, the process using the Wiener filter will be described as an example, but the image processing unit 7 may execute deconvolution by other processes. The deconvolution of each detection unit by the Wiener filter is represented by the following formula (9).

In Expression (9), Obj ^to (r, k _s ) are distributions of the fluorescent substance estimated for each detection unit 6a of the detection device 6 (hereinafter, referred to as estimated fluorescent substance distributions). w is a Wiener parameter for suppressing noise. By this processing, the estimated fluorescent substance distributions Obj 1 ^to (r, k _s ) are substantially the same in the two or more detection units 6a of the detection device 6. That is, the above processing corrects the shape collapse and the positional deviation of the effective PSF for each detector 6a, and the effective PSF for each detector 6a becomes substantially the same. The image processing unit 7, the process shown in the following equation ^(10), the apodization Obj ~ (r, _{k s),} adding the spectrum in the detection section 6a of the detection device 6, the super-resolution image _{I SR} ( to generate a r _s).

In Expression (10), A(k _s ) is an apodization function for suppressing the negative value of the image, and multiplying Obj ^˜ (r, k _s ) by A(k _s ) is called apodization. The functional form of A(k _s ) is designed to suppress the negative value of the image by theoretical calculation or simulation. _{Further, F} ^{ks -1} is the inverse Fourier transform relates _{k s.} The image processing unit 7 performs the inverse Fourier transform after adding the spectra of the detection units 6a, but may add the images after performing the inverse Fourier transform. In the processing of Expressions (9) and (10), the image processing unit 7 independently deconvolves each detection unit 6a and then adds the images to each detection unit 6a. The image processing section 7 may collectively deconvolute two or more detection sections 6a as in the following Expression (11).

FIG. 8 is a flowchart showing an observation method according to the second embodiment. The processing from step S11 to step S14 is the same as the processing from step S1 to step S4 in FIG. 6, and therefore description thereof will be omitted. In step S15, the image processing unit 7 Fourier transforms the detection result of each detection unit. In step S16, the image processing unit 7 executes deconvolution. In step S17, the image processing unit 7 executes apodization. Apodization may be part of the deconvolution process. In step S18, the image processing unit 7 adds the images of the detection units 6a using the result of the deconvolution. In step S19, the image processing unit 7 performs an inverse Fourier transform on the first image (eg, Fourier image) obtained in step S18 to generate a second image (eg, super-resolution image).

According to the second embodiment, the image processing unit 7 performs Fourier transform on the detection result of each detection unit and executes deconvolution, thereby aligning the effective PSFs of the images obtained for each detection unit 6a. Therefore, the image processing unit 7 adds the images of the respective detection units 6a using the result of the deconvolution to generate an image with high resolution (eg, super-resolution image) while ensuring S/N. be able to. Further, the fluorescent substance contained in the sample S is multiphoton excited by the excitation light L1. As a result, similar to the first embodiment, the image data on the sample surface Sa set in an arbitrary portion including the inside of the sample S can be acquired with high accuracy.

Note that the image processing unit 7 may change the range of the detection unit 6a to be added, as described in the first embodiment. Further, the image processing unit 7 may improve the resolution one-dimensionally or two-dimensionally as described in the first embodiment.

[Third Embodiment]
A third embodiment will be described. In the present embodiment, the same components as those in the above-described embodiment will be designated by the same reference numerals, and the description thereof will be omitted or simplified. FIG. 9 is a diagram showing processing of the image processing unit of the microscope according to the third embodiment. For each part of the microscope, refer to FIG. 1 or FIG. 4 as appropriate.

In the present embodiment, the image processing unit 7 (see FIG. 4) corrects the collapse of the shape of the effective PSF for each detection unit 6a by image processing different from that in the second embodiment. FIG. 9A is a PSF before image processing according to the present embodiment, which is the same as FIG. 5A. In FIG. 9A, the distance between the detection units 6a of the detection device 6 (see FIG. 4) does not match the distance between the interference fringes L2. In this case, the shape of the effective PSF (solid line) of the image obtained for each detection unit 6a is deformed depending on the position of the detector 6a, as described in the above equation (5). The image processing unit 7 sets the interference fringes so that the peak position of the partial distribution of intensity distribution of the interference fringes L2 (eg, Q2b shown in the graph on the left side of FIG. 9A) matches the position of the detection unit 6a. The phase of the intensity distribution of L2 is effectively shifted by image processing. This process is appropriately referred to as image processing phase shift processing. Further, the phase shift by the image processing phase shift processing is appropriately referred to as an image processing phase shift. The amount of phase shift by image processing phase shift processing is appropriately referred to as image processing phase shift amount.

FIG. 9B shows an effective PSF for each detection unit 6a after the image processing phase shift processing. In the graph on the left side of FIG. 9B, the distribution Q2f is image processing phase-shifted so that the peak position X2b of the distribution Q2b of FIG. 9A coincides with the position X1b of the detection unit 6a. Distribution. The peak position X2f of the distribution Q2f substantially coincides with the position X1b of the detection unit 6a. Reference numeral Q1f is a distribution corresponding to the effective PSF obtained from the distribution Q2f in which the phase is subjected to the image processing phase shift and the detected PSF (distribution Q3b) of the detection unit 6a arranged at the position X1b. In the distribution Q1f, the collapse of the shape of the effective PSF is reduced.

Further, in the graph on the right side of FIG. 9B, the distribution Q2g has the phase of the distribution Q2 set to the image processing phase so that the peak position X2c of the distribution Q2c of FIG. 9A matches the position X1c of the detection unit 6a. This is the shifted distribution. The peak position X2g of the distribution Q2g substantially coincides with the position X1c of the detection unit 6a. Reference numeral Q1g is a distribution corresponding to the effective PSF obtained from the distribution Q2g in which the phase is subjected to the image processing phase shift and the detected PSF (distribution Q3c) of the detection unit 6a arranged at the position X1c. In the distribution Q1g, the collapse of the shape of the effective PSF is reduced.

By the image processing as described above, the shape of the effective PSF (solid line) for each detection unit 6a is corrected so as to be substantially the same. The image processing unit 7 generates an image by using an image for each detection unit 6a having an effective PSF corrected to have substantially the same shape.

Hereinafter, the processing flow of the image processing unit 7 will be described in more detail. The image I(r,r _s ) obtained by the detection device 6 is represented by the above equation (3). Further, here, the case of two-photon excitation (the case of n = 2 in the equation (3)) will be described as an example. Substituting ill (r) shown in the above equation (2) and n = 2 indicating the case of two-photon excitation into the equation (3), the following equation (12) is obtained.

In equation (12), φ indicates the initial phase of the interference fringe L2. The term of cos(2πk ₀ ·r _s +φ) in Expression (12) has the same period as the period of the interference fringes formed by the excitation light. On the other hand, the term of cos(4πk ₀ ·r _s +2φ) in the equation (12) has a half cycle of the interference fringe cycle. In the case of two-photon excitation, up to the second harmonic component of the period of the interference fringes formed by the excitation light is generated. In the case of n-photon excitation, up to the nth harmonic component of the period of the interference fringes formed by the excitation light is generated. The image processing unit 7 changes the phase of the interference fringe L2 by image processing according to the detector coordinates, and aligns the shapes of the effective PSFs. The microscope 1 acquires the four-dimensional image data I(r,r _s ) as described in Expression (3). The image processing unit 7 performs a four-dimensional Fourier transform on I(r,r _s ). The four-dimensional data of the frequency space obtained by the Fourier transform is represented by I ^~ (k, k _s ).

By performing Fourier transform on the above equation (12) for r and r _s , it can be seen that I 1 ^to (k, k _s ) are represented by the following equation (13).

Here, I ₀ ^to (k, k _s ) of the first term on the right side of the equation (13) are referred to as a zero-order component. I ₀ ^to (k, k _s ) are defined as the following equation (14).

I ₊₁ ^to (k, k _s ) of the second term on the right side of the equation (13) are called +1 order components. I _{+ 1} ^to (k, k _s ) are defined by the following equation (15).

I _-1 ^to (k, k _s ) of the third term on the right side of the equation (13) are called -1st order components. I _-1 ^to (k, k _s ) are defined as the following equation (16).

_I ⁺² ~ the fourth term on the right-hand side of equation (13) to (k, _{k s)} is called a +2 order component. I ₊₂ ^to (k, k _s ) are defined by the following equation (17).

_I ^-2 ~ the right side fifth term of formula (13) to (k, _{k s)} is referred to as a minus second-order component. ^{_{I -2 ~ (k, k s}} ) is defined as the following equation (18).

In the formulas (14) to (18), the OTF _det (k) is a Fourier transform of the PSF _det (r) and represents the OTF of the detection optical system 5. Further, OTF _'ill (k) is obtained by Fourier transform of the PSF ² _ill (r). Obj ^~ _(k s) are those that Obj a _{(r s)} obtained by Fourier transformation.

The cutoff frequency of OTF _det (k) is given by 2k _NA ^em . Further, the cut-off frequency of OTF _'ill (k) is given by the 4σk _NA ^ex. Therefore, I ₀ ^to (k, k _s ) have values only within the region satisfying the condition of the following equation (19). A region that satisfies the condition of the following Expression (19) is appropriately referred to as a zero-order component region AR1a.

I ₊₁ ^to (k, k _s ) have values only within the region satisfying the condition of the following equation (20). A region that satisfies the condition of the following Expression (20) is appropriately referred to as a +1st order component region AR1b.

I _-1 ^to (k, k _s ) have values only within the region satisfying the condition of the following equation (21). A region that satisfies the condition of the following expression (21) is appropriately called a -1st order component region AR1c.

^{_{I +2 ~ (k, k s}} ) has a value only in satisfying the region of Formula (22) below. A region that satisfies the condition of the following Expression (22) is appropriately referred to as a +second-order component region AR1d.

I _-2 ^to (k, k _s ) have values only within the region satisfying the condition of the following equation (23). A region that satisfies the condition of the following Expression (23) is appropriately referred to as a −second-order component region AR1e.

The image processing section ^7, I-(k, _{k s)} from, by extracting the information of any of satisfying area of formula (19) to Formula ^{_{(23), I 0 ~ (}} k, k s) , I ₊₁ ^to (k, k _s ), I ₋₁ ^to (k, k _s ), I ₊₂ ^to (k, k _s ), and I ₋₂ ^to (k, k _s ). I 0 ^to (k, k _s ) to I ₀ ^to (k, k _s ), I ₊₁ ^to (k, k _s ), I ₋₁ ^to (k, k _s ), I ₊₂ ^to (k, k _s ), and _{^{I -2 ~ (k, k s}} ) a process of separating, appropriately referred to as a component separation. When a region satisfying any of the conditions of the formulas (19) to (23) is illustrated, it has the shape shown in FIG. 10, for example.

FIG. 10 is a diagram showing a region of the frequency space used for component separation in the third embodiment. Here, the case where the openings 15a and 15b of the mask 15 (see FIG. 2) are circular will be described. The opening of the mask 15 may have a shape other than a circular shape. The range of the area AR1a of the 0th order component, the area AR1b of the +first order component, the area AR1c of the −1st order component, the area AR1d of the +second order component, and the area AR1e of the −second order component is when the opening of mask 15 is circular. In any case where the opening of the mask 15 has a shape other than a circular shape, it can be obtained by numerical simulation, theoretical calculation, or the like.

FIG. 10A shows each region on the k _{xs -kys} plane. The 0th-order component area AR1a, the +1st-order component area AR1b, the −1st-order component area AR1c, the +second-order component area AR1d, and the −second-order component area AR1e are circular areas. The 0th-order component area AR1a, the +1st-order component area AR1b, the −1st-order component area AR1c, the +second-order component area AR1d, and the −second-order component area AR1e have the same diameter. The diameter of the 0th-order component region AR1a is 8σk _NA ^ex . The region AR1a of the 0th-order component is a region centered on the origin. The area AR1b of the +1st order component, the area AR1c of the −1st order component, the area AR1d of the +second order component, and the area AR1e of the −second order component are areas on the axis of which the center is k _xs . The distance A between the center and the origin of the region AR1c of the -1st order component is 2 (1-σ) k _NA ^ex . The +1st-order component region AR1b is a region symmetric with respect to the -1st-order component region AR1c with respect to the 0th-order component region AR1a. The distance B between the center of the area AR1e of the −second-order component and the origin is twice the distance A, which is 4(1−σ)k _NA ^ex . The + secondary component region AR1d is a region symmetric with respect to the second-order component region AR1e with respect to the zero-order component region AR1a.

FIG. 10B shows each region on the k _xs −k _x plane. The 0th-order component area AR1a, the +1st-order component area AR1b, the −1st-order component area AR1c, the +second-order component area AR1d, and the −second-order component area AR1e are parallelogram areas, respectively.

The image processing unit 7 sets the region of the frequency space in the component separation based on the light intensity distribution of the excitation light in the sample S. For example, the image processing unit 7 sets a plurality of regions that do not overlap each other based on the square of the electric field intensity ill(r) of the excitation light on the sample surface Sa as the light intensity distribution of the excitation light on the sample S. The plurality of regions described above include five or more regions that do not overlap with each other. For example, the plurality of areas include the area AR1a of FIG. 10 as the first area, the area AR1b of FIG. 10 as the second area, the area AR1c of FIG. 10 as the third area, the area AR1d of FIG. 10 as the fourth area, and the area AR1d of FIG. The region AR1e of FIG. 10 is included as the five regions. The image processing unit 7 selects, from the data in the frequency space, data belonging to the first area (area AR1a), data belonging to the second area (area AR1b), data belonging to the third area (area AR1c), and the fourth area ( The components are separated by extracting the data belonging to the area AR1d) and the data belonging to the fifth area (area AR1e). Not only in the case of two-photon excitation but also in the case of n-photon excitation, the image processing unit 7 determines the electric field intensity ill(r) of the excitation light on the sample surface Sa as the light intensity distribution of the excitation light on the sample S. It is possible to set a plurality of regions that do not overlap each other based on the nth power and perform component separation.

The image processing unit 7 uses I ₀ ^to (k, k _s ), I ₊₁ ^to (k, k _s ), I ₋₁ ^to (k, k _s ), I ₊₂ ^to (k, k _s ), I _−2. each ^{~ (k,} k _s) to, by inverse Fourier transform of the four-dimensional, calculates the image data of real space. Hereinafter, image data obtained by performing an inverse Fourier transform on I ₀ ^to (k, k _s ) will be represented by I ₀ (r, r _s ). Image data obtained by inverse Fourier transforming I ₊₁ ^to (k, k _s ) is represented by I ₊₁ (r, r _s ). Image data obtained by inverse Fourier transforming I ₋₁ ^to (k, k _s ) is represented by I ₋₁ (r, r _s ). Image data obtained by inverse Fourier transforming I ₊₂ ^to (k, k _s ) is represented by I ₊₂ (r, r _s ). Image data obtained by inverse Fourier transforming I ₋₂ ^to (k,k _s ) is represented by I ₋₂ (r,r _s ). The image processing unit 7 uses I ₀ (r,r _s ), I ₊₁ (r,r _s ), I ₋₁ (r,r _s ), I ₊₂ (r,r _s ), and I ₋₂ (r,r ). _s ) is subjected to the calculations shown in the following formulas (24A) to (24E).

In the formulas (24A) to (24E), ψ (r) represents the position of the detection unit 6a of the detection device 6, that is, the image processing phase shift amount for each detector coordinate r. The image processing phase shift amount ψ (r) is determined, for example, as follows. The image processing unit 7 calculates the positional deviation amount of the signal detected at the detector coordinate r. The image processing unit 7, for example, by simulation in _advance, by obtaining the peak position of the obtained function _{PSF det} and (r + r _s) by the product of ^PSF ₂ ill _{(r s),} calculates the position deviation amount of the .. Here, the misalignment of the effective PSF may be considered to be proportional to the detector coordinates r, and if the parameter representing the degree of misalignment is β, the amount of misalignment of the effective PSF can be represented by r / β.

The value of β _may be calculated from the peak position of the function obtained by the product of the _{PSF det} (r + r _s) and _PSF ill _{(r s),} may be calculated by other numerical simulations. When β is determined, the image processing phase shift amount ψ (r) according to the detector coordinates r is determined. Image processing the phase shift of the interference fringe L2 [psi (r) _is the peak position of the function obtained by the product of the _{PSF det} (r + r _s) and _PSF ill _{(r s),} and the peak position of the interference fringes L2 is Determined to match. By such processing, the image processing phase shift amount ψ(r) becomes, for example, ψ(r)=−2πk ₀ ·r/β−φ. The value of the initial phase φ may be a value measured in advance using fluorescent beads or a value estimated from an observed image. The image processing unit 7 determines the amount of phase conversion (image processing phase shift amount) based on the light intensity distribution of the excitation light in the sample S.

When the image processing phase shift amount ψ(r) is determined, the image processing unit 7 calculates the sum of the calculation results of the above formulas (24A) to (24E) as shown in the following formula (25).

The above equation (25) I obtained from the calculation of _'(r, r _s) is broken in the shape of the effective PSF for each position of the detection unit 6a (detector coordinates r) is corrected, substantially the same shape of the effective PSF It becomes an image that became. The image processing section _{7, I '(r, r s} ) with respect to correct the positional deviation of the effective PSF for each detected portion 6a of the detector 6. Thereby, the effective PSFs can be made substantially the same in the two or more detection units 6a of the detection device 6. The image processing unit 7, by adding the image for each detector 6a the effective PSF is corrected so as to be substantially the same, to produce a super-resolution image I _{SR (r s).} This series of processing is performed based on the equation (26).

In Expression (26), PH(r) is a pinhole function defined by the following Expression (27).

By adjusting the value of r _PH , the signal amount and the sectioning effect can be adjusted. Increasing the value of r _PH increases the amount of signal. A smaller value of r _PH improves the sectioning ability. Although the case of two-photon excitation has been described as an example, it is possible to perform image processing based on the same concept in the case of n-photon excitation.

In the present embodiment, the scan interval and the interval of the detection unit 6a of the detection device 6 may be set based on the cutoff frequency and the Nyquist theorem. When n-photon excitation is used, the scan interval may be set to λ _ex /(8nNA) or less in the periodic direction of the interference fringes. When n-photon excitation is used, the scan interval may be set to λ _ex /(4nNA) or less in the direction perpendicular to the periodic direction of the interference fringes. Further, the interval between the detection units 6a of the detection device 6 may be set to λ _em /4NA or less.

FIG. 11 is a flowchart showing an observation method according to the third embodiment. Since the processing of steps S21 to S24 is the same as the processing of steps S1 to S4 of FIG. 6, the description thereof will be omitted. In step S25, the image processing unit 7 Fourier transforms at least a part of the detection results of the plurality of detection units 6a. In step S25, the image processing section 7 performs Fourier transformation of 4-dimensional relative I _(r, r _s). In step S26, the image processing unit 7 separates the components in the frequency space. The image processing unit 7 separates the components of the frequency space obtained by the Fourier transform into each region of the frequency space. In step S27, the image processing unit 7 performs an inverse Fourier transform on the separated components. In step S28, the image processing unit 7 executes the image processing phase shift processing. The image processing unit 7 corrects the positional deviation of the effective PSF in step S29. In step S30, the image processing unit 7 generates an image (for example, a super-resolution image) by adding the images obtained by correcting the positional deviation in step S29.

As described above, the image processing unit 7 according to the present embodiment converts the phase of at least a part of the data obtained by the component separation to generate an image. In the above description, the image processing unit 7 executes the phase shift process on the data in the real space. That is, the image processing unit 7 uses, as the data obtained by the component separation, the data (the data on the real space) obtained by converting the data on the component separation (the data on the frequency space) into the data on the real space by the inverse Fourier transform. .. The image processing unit 7 may perform the phase shift process in the frequency space on the data in the frequency space in which the components have been separated.

According to the third embodiment, the image processing unit 7 executes the phase shift process to align the effective PSFs of the images obtained by the detection units 6a. Therefore, the image processing unit 7 adds the images of the respective detection units 6a obtained by executing the phase shift process to obtain an image with high resolution (eg, super-resolution image) while ensuring S/N. Can be generated. The fluorescent substance contained in the sample S is multiphoton excited by the excitation light L1. As a result, similar to the first embodiment, the image data on the sample surface Sa set in an arbitrary portion including the inside of the sample S can be acquired with high accuracy.

[Fourth Embodiment]
A fourth embodiment will be described. In the present embodiment, the same components as those in the above-described embodiment will be designated by the same reference numerals, and the description thereof will be omitted or simplified. In the present embodiment, the image processing unit 7 (see FIG. 4) performs the component separation described in the third embodiment, and then performs deconvolution on the separated components to generate an image.

Rewriting formula (13), formula (28) is obtained.

In Expression (28), OTF ₀ (k, k _s ), OTF ₊₁ (k, k _s ), OTF ₋₁ (k, k _s ), OTF ₊₂ (k, k _s ), OTF ₋₂ (k, k _s ) _s ) is represented by the following formulas (29A) to (29E).

The image processing unit 7 includes OTF ₀ (k, k _s ), OTF ₊₁ (k, k _s ), OTF ₋₁ (k, k _s ), OTF ₊₂ (k, k _s ), OTF ₋₂ (k, k _s ). Deconvolution is performed using each measured value or estimated value of _{s 2} ). There are various methods of deconvolution, such as the Wiener filter and the Richardson-Lucy method. Here, processing using a Wiener filter will be described as an example of deconvolution, but deconvolution using another method may be used. With respect to the above equation (28), deconvolution by the Wiener filter is expressed by the following equation (30A) and equation (30B).

In Expression (30A), w is a Wiener parameter for suppressing noise. In the formula _{(30B), A (k s} ) is the apodization function for suppressing the negative value of the image. _{Further, F} ^{ks -1} is the inverse Fourier transform relates _{k s.} The image processing section 7 generates the super-resolved image I _{SR (r s)} by using a de-convolution results described above.

FIG. 12 is a flowchart showing an observation method according to the fourth embodiment. The processing from step S31 to step S34 is similar to the processing from step S1 to step S4 in FIG. 6, and therefore description thereof will be omitted. In step S35, the image processing unit 7 Fourier transforms the detection result. Further, in step S36, the image processing unit 7 separates the components in the frequency space. In step S37, the image processing unit 7 performs deconvolution using the components separated by the process of step S36. In step S38, the image processing unit 7 performs apodization. In step S39, the image processing unit 7 performs an inverse Fourier transform on the data obtained by the deconvolution and the apodization. The image processing unit 7 uses the data obtained by the inverse Fourier transform to generate an image (for example, a super-resolution image).

As described above, the image processing unit 7 according to the present embodiment executes component separation, deconvolution, and apodization in the frequency space, converts the data obtained by these processes into the data in the real space, and forms the image. To generate. In the present embodiment, the image processing unit 7 may generate an image without performing the process of correcting the positional deviation by making the effective PSFs of the detection units 6a of the detection devices 6 substantially match.

According to the fourth embodiment, the image processing unit 7 executes the component separation, deconvolution, and apodization in the frequency space to align the effective PSFs of the images obtained by the detection units 6a. Then, the image processing unit 7 converts the data obtained by these processes into data in the real space to generate an image, thereby ensuring an S/N and an image with high resolution (eg, super-resolution). Image) can be generated. The fluorescent substance contained in the sample S is multiphoton excited by the excitation light L1. As a result, similar to the first embodiment, the image data on the sample surface Sa set in an arbitrary portion including the inside of the sample S can be acquired with high accuracy.

In the present embodiment, the scan interval and the interval of the detection unit 6a of the detection device 6 may be set based on the cutoff frequency and the Nyquist theorem. In the case of n photon excitation, the scan interval may be set to λ _ex / (8 nNA) or less in the periodic direction of the interference fringes. In the case of n-photon excitation, the scan interval may be set to λ _ex /(4nNA) or less in the direction perpendicular to the periodic direction of the interference fringes. Further, the interval between the detection units 6a of the detection device 6 may be set to λ _em /4NA or less.

The image processing unit 7 may set the range to be integrated with respect to the above k to the range of the entire space or to a part of the range of the entire space. Further, the image processing unit 7 uses I ₀ ^to (k, k _s ), I ₊₁ ^to (k, k _s ), I ₋₁ ^to (k, k _s ), and I ₊₂ ^to (k, k _s ) by Fourier transform. ), I _-2 ^to (k, k _s ) may be limited in the range of r. The image processing unit 7 also includes OTF ₀ (k, k _s ), OTF ₊₁ (k, k _s ), OTF ₋₁ (k, k _s ), OTF ₊₂ (k, k _s ), and OTF − ₂ ( k, k _s ) may be data obtained in advance by measurement using fluorescent beads or numerical simulation using design values, or obtained from the result of detection of fluorescence from the sample S by the detection device 6. Data (eg, estimates) may be used.

[Fifth Embodiment]
A fifth embodiment will be described. In the present embodiment, the same components as those in the above-described embodiment will be designated by the same reference numerals, and the description thereof will be omitted or simplified. FIG. 13 is a diagram showing a microscope 101 according to the fifth embodiment. In the fifth embodiment, the detection device 106 includes a line sensor (line detector) in which a plurality of detection units 6a are arranged one-dimensionally. The plurality of detection units 6a are arranged in one direction in the detection device 106. The detection device 106 is arranged at a position optically conjugate with the sample surface Sa. The direction in which the plurality of detection units 6a are arranged (hereinafter referred to as the arrangement direction) is set to the direction corresponding to the periodic direction of the interference fringes L2. For example, in FIG. 13, the periodic direction of the interference fringes is the X direction, and the array direction of the plurality of detection units 6a is set to the Xb direction corresponding to the X direction.

The microscope 101 according to the fifth embodiment includes a λ/2 wavelength plate 30 and an optical path rotating unit 31 that rotates the optical path around the optical axis. The λ/2 wave plate 30 rotates the polarized light passing through the optical path rotating unit 31 based on the rotation angle of the optical path by the optical path rotating unit 31. The optical path rotating unit 31 is arranged in the optical path between the mask 15 and the sample S in the illumination optical system 4. The optical path rotating unit 31 is arranged, for example, at a position where the excitation light L1 becomes substantially parallel light in the optical path of the illumination optical system 4. The optical path rotating unit 31 is arranged, for example, at a position where the excitation light L1 passes through in the illumination optical system 4 and the fluorescence L3 passes through in the detection optical system 5. The optical path rotating unit 31 is arranged in the optical path between the dichroic mirror 16 and the sample S, for example. The λ/2 wave plate 30 may be arranged on the same side as the sample S with respect to the optical path rotating unit 31, or on the side opposite to the sample S with respect to the optical path rotating unit 31 (eg, on the same side as the excitation light source). ).

The optical path rotating unit 31 is, for example, an image rotator such as a Dove prism. The optical path rotating unit 31 is provided rotatably around the optical axis of the illumination optical system 4. The optical path rotating unit 31 is driven by the driving unit 32 and rotates. When a Dove prism is used as the optical path rotating unit 31, when the Dove prism is rotated by θ° around the optical axis of the illumination optical system 4, the optical path on the light emission side (the sample S side) from the Dove prism is directed to the Dove prism. The optical path on the light incident side (light source 3 side) is rotated by 2×θ° around the optical axis of the illumination optical system 4. As a result, the incident surface of the excitation light L1 on the sample S rotates 2×θ° around the Z direction, and the periodic direction of the interference fringes L2 rotates 2×θ° around the Z direction. For example, when the periodic direction of the interference fringe L2 is changed by 90 °, the driving unit 32 rotates the optical path rotating unit 31 by 45 ° around the optical axis of the illumination optical system 4. As described above, the optical path rotating portion 31 is included in the fringe direction changing portion that changes the direction of the interference fringes with respect to the sample.

The λ/2 wave plate 30 is provided so as to be rotatable around the optical axis of the illumination optical system 4. The λ / 2 wave plate 30 rotates in conjunction with the optical path rotating portion 31. The λ / 2 wave plate 30 rotates by an angle determined based on the rotation angle of the optical path rotating portion 31. For example, the λ/2 wavelength plate 30 is fixed (eg, integrated) with the optical path rotating unit 31 and rotates together with the optical path rotating unit 31. In this case, the λ/2 wave plate 30 rotates by the same angle as the rotation angle of the optical path rotating unit 31.

When the λ/2 wave plate 30 is rotated by θ° around the optical axis of the illumination optical system 4, the polarization direction of the excitation light L1 is different from the polarization direction on the light incident side (the light source 3 side). Rotate 2 × θ ° around the optical axis of. As a result, the polarization state of the excitation light L1 when entering the sample S becomes S-polarized.

The optical path rotating unit 31 of FIG. 13 is also included in the image rotating unit. The image rotation unit rotates an image of the sample S (eg, an image of fluorescence from the sample S) with respect to the plurality of detection units 6a around the optical axis of the detection optical system 5. That is, the fringe direction changing portion and the image rotating portion include the optical path rotating portion 31 as the same member (optical member). The optical path rotating unit 31 is arranged at a position on the optical path of the illumination optical system 4 where fluorescence enters. The image rotating unit rotates the fluorescence image by the optical path rotating unit 31. The optical path rotation unit 31 adjusts the cycle direction of the interference fringes L2 with respect to the arrangement direction of the plurality of detection units 6a in the detection device 6. When the Dove prism is used as the optical path rotating unit 31, when the Dove prism is rotated by θ° around the optical axis of the illumination optical system 4, the cycle direction of the interference fringes L2 is rotated by 2×θ° around the Z direction. Then, the optical path of the fluorescence L3 from the sample S rotates −2×θ° on the light emitting side (detector 106 side) with respect to the light incident side (sample S side) on the Dove prism.

When the dub prism is rotated, the optical path of light toward the sample S through the dub prism is rotated, and the periodic direction of the interference fringe L2 with respect to the sample S changes. The optical path of the light traveling from the sample S to the detection device 106 via the Dove prism rotates in the opposite direction to the optical path of the light traveling to the sample S by the same angle. Therefore, when the images of the plurality of detection units 6a (eg, line detectors) in the detection device 106 are projected onto the sample surface Sa through the detection optical system 5, the direction in which the plurality of detection units 6a are arranged and the periodic direction of the interference fringes. Always matches even when the periodic direction of the interference fringes is changed by the Dove prism. Therefore, the detection device 106 can detect the fluorescence L3 similarly before and after the change of the periodic direction of the interference fringe L2.

The resulting image data I (x, _{r s)} is a three-dimensional data with detectors coordinates x and scan coordinate _{r s = (x s, y} s) of the independent variable. In the first to fourth embodiments, since the detection device 6 includes the plurality of detection units 6a arranged two-dimensionally, the detector coordinates have a two-dimensional coordinate system. In the fifth embodiment, since the detection device 106 includes the plurality of detection units 6a arranged one-dimensionally, the detector coordinates have a one-dimensional coordinate system. In the image processing described in the first to fourth embodiments, the processing relating to detector coordinates is image processing corresponding to a one-dimensional coordinate system (eg, Fourier transform for detector coordinates in the third embodiment is one-dimensional Fourier transform). It can be converted). Therefore, the image processing described in the first to fourth embodiments can be applied to the fifth embodiment as well. The image processing unit 7 generates an image (for example, a super-resolution image) by one of the processes described in the first to fourth embodiments based on the detection result of the detection device 106.

In the microscope 1 according to the first embodiment, the driving unit 22 rotates the mask 15 to change the cycle direction of the interference fringes L2. However, the interference fringes L2 are changed by the optical path rotating unit 31 (for example, the Dove prism). The periodic direction of may be changed. The fringe direction changing unit that changes the cycle direction of the interference fringes L2 may be different from both the drive unit 22 and the optical path rotating unit 31. For example, the stage 2 may be rotatably provided around the Z direction, and the rotation may change the direction of the interference fringes L2 with respect to the sample S. In this case, the stage 2 is included in the fringe direction changing portion that changes the direction of the interference fringe L2 with respect to the sample S.

The microscope 101' shown in FIG. 14 is a modification of the microscope 101 according to the fifth embodiment, and the position at which the optical path rotating unit 31 is provided is different from that of the microscope 101 shown in FIG. In the microscope 101 ′ shown in FIG. 14, the stripe direction changing unit is the same as in the first embodiment and includes the mask 15 and the driving unit 22. In the microscope 101 shown in FIG. 13, the optical path rotating unit 31 serves as both the stripe direction changing unit and the image rotating unit, but in the microscope 101′ shown in FIG. 14, the optical path rotating unit 31 is provided separately from the stripe direction changing unit. .. In the microscope 101' shown in FIG. 14, the optical path rotating unit 31 is arranged at a position that does not overlap the optical path of the illumination optical system 4 in the optical path of the detection optical system 5. In other words, the optical path rotating unit 31 is arranged at a position where the excitation light L1 does not enter and the fluorescence L3 enters. The optical path rotating unit 31 is arranged in the optical path between the dichroic mirror 16 and the detection device 106.

In the microscope 101' shown in FIG. 14, the driving unit 22 rotates the mask 15 and the polarizer 14 to change the period direction of the interference fringe L2. The drive unit 32 rotates the optical path rotation unit 31 by an angle determined based on the rotation angle of at least one of the mask 15 and the polarizer 14. In the microscope 101 ′ shown in FIG. 14, the drive unit 32 rotates the optical path rotation unit 31 to match the direction of the image projected on the detection device 106 with the direction in which the plurality of detection units 6 a are arranged.

According to the fifth embodiment, it is possible to align the effective PSFs of the images obtained for each detection unit 6a by any of the processes described in the first to fourth embodiments. Therefore, it is possible to generate an image (for example, a super-resolution image) with high resolution while ensuring S/N. The fluorescent substance contained in the sample S is multiphoton excited by the excitation light L1. As a result, similar to the first to fourth embodiments, the image data on the sample surface Sa set in an arbitrary portion including the inside of the sample S can be acquired with high accuracy.

[Sixth Embodiment]
The sixth embodiment will be described. In the present embodiment, the same components as those in the above-described embodiment will be designated by the same reference numerals, and the description thereof will be omitted or simplified. FIG. 15 is a diagram showing a microscope 201 according to the sixth embodiment. The microscope 201 according to the sixth embodiment includes a light-shielding member 33. The light shielding member 33 is arranged at a position optically conjugate with the sample surface Sa or in the vicinity thereof. In the sixth embodiment, the detection device 106 includes a line sensor in which a plurality of detection units 6a are arranged one-dimensionally, as in the fifth embodiment. The detection device 106 is arranged at a position optically conjugate with the sample surface Sa, and the light shielding member 33 is arranged in the vicinity of the detection device 106. The light blocking member 33 may be arranged at a position conjugate with the sample surface Sa or in the vicinity thereof.

The light shielding member 33 has an opening 33a through which the fluorescence L3 passes, and shields the fluorescence L3 around the opening 33a. The opening 33a extends in the arrangement direction (Xb direction) of the plurality of detection units 6a in the detection device 106. The opening 33a is, for example, a rectangular slit. The light shielding member 33 is arranged such that the long side of the opening 33a is substantially parallel to the arrangement direction of the plurality of detection units 6a. One or both of the size and the shape of the opening 33a may be variable in the light blocking member 33. For example, the light blocking member 33 may be a mechanical diaphragm having a variable light blocking region or a spatial light modulator (SLM). .. One or both of the size and the shape of the opening 33a may be fixed.

The detection device 106 detects the fluorescence L3 that has passed through the opening 33a of the light-shielding member 33. As described in the fifth embodiment, in the image processing described in the first embodiment to the fourth embodiment, the processing regarding the detector coordinates can be the image processing corresponding to the one-dimensional coordinate system. Therefore, the image processing described in the first to fourth embodiments can be applied to the sixth embodiment as well as the fifth embodiment. The image processing unit 7 generates an image (for example, a super-resolution image) by one of the processes described in the first to fourth embodiments based on the detection result of the detection device 106.

According to the sixth embodiment, it is possible to align the effective PSFs of the images obtained by the detection units 6a by any of the processes described in the first to fourth embodiments. Therefore, it is possible to generate an image (for example, a super-resolution image) with high resolution while ensuring S/N. The fluorescent substance contained in the sample S is multiphoton excited by the excitation light L1. As a result, similar to the first to fourth embodiments, the image data on the sample surface Sa set in an arbitrary portion including the inside of the sample S can be acquired with high accuracy.

[Seventh Embodiment]
The seventh embodiment will be described. In the present embodiment, the same components as those in the above-described embodiment will be designated by the same reference numerals, and the description thereof will be omitted or simplified. FIG. 16 is a diagram showing a microscope 301 according to a seventh embodiment. The microscope 301 according to the seventh embodiment includes a drive unit 22 and a drive unit 34. The drive unit 22 is similar to that of the first embodiment. The drive unit 22 rotates the mask 15 to change the cycle direction of the interference fringe L2. The driving unit 22 is included in the fringe direction changing unit that changes the direction of the interference fringe L2 with respect to the sample S.

In the seventh embodiment, the detection device 106 includes a line sensor in which a plurality of detection units 6a are arranged one-dimensionally, as in the sixth embodiment. The detection device 106 is arranged at a position optically conjugate with the sample surface Sa, and the light shielding member 33 is arranged near the detection device 106. In a seventh embodiment, the detector 106 is rotatable around the Zb direction. The drive unit 34 rotates the detection device 106 around the Zb direction. The drive unit 34 rotates the detection device 106 so that the arrangement direction of the detection units 6a in the detection device 106 corresponds to the cycle direction of the interference fringes L2. For example, when the drive unit 22 rotates the mask 15 by 90 °, the periodic direction of the interference fringe L2 changes by 90 °, so that the drive unit 34 rotates the detection device 106 by 90 °.

Further, the drive unit 34 rotates the light blocking member 33 so that the relative position between the detection device 106 and the light blocking member 33 is maintained. For example, the light blocking member 33 and the detection device 106 are integrated, and the drive unit 34 integrally rotates the light blocking member 33 and the detection device 106.

The detection device 106 detects the fluorescence L3 that has passed through the opening 33a of the light shielding member 33. As described in the fifth embodiment, in the image processing described in the first embodiment to the fourth embodiment, the processing regarding the detector coordinates can be the image processing corresponding to the one-dimensional coordinate system. Therefore, the image processing described in the first to fourth embodiments can be applied to the case of the seventh embodiment as well as the fifth embodiment. The image processing unit 7 generates an image (for example, a super-resolution image) by one of the processes described in the first to fourth embodiments based on the detection result of the detection device 106.

According to the seventh embodiment, the effective PSFs of the images obtained for each detection unit 6a can be aligned by any of the processes described in the first to fourth embodiments. Therefore, it is possible to generate an image (for example, a super-resolution image) with high resolution while ensuring S/N. The fluorescent substance contained in the sample S is multiphoton excited by the excitation light L1. As a result, similar to the first to fourth embodiments, the image data on the sample surface Sa set in an arbitrary portion including the inside of the sample S can be acquired with high accuracy.

In the above-described seventh embodiment, the microscope may include the optical path rotating unit 31 shown in FIG. 14 instead of the drive unit 34 that rotates the detection device 106. In the above-described seventh embodiment, the microscope need not include the light shielding member 33 as shown in FIG.

[Eighth Embodiment]
The eighth embodiment will be described. In the present embodiment, the same components as those in the above-described embodiment will be designated by the same reference numerals, and the description thereof will be omitted or simplified. FIG. 17 is a diagram showing a microscope 401 according to the eighth embodiment. In the above-described embodiment, the example in which the illuminated pupil is divided into two poles (two regions) on the pupil surface P0 (see FIG. 2C) has been described, but the illuminated pupil may be in another form. Here, a form in which the illumination pupil is divided into four poles (four regions) on the pupil plane will be described.

The illumination optical system 404 according to the eighth embodiment includes a collimator lens 12, a λ/2 wavelength plate 35, a polarization separation element 36, a mirror 37, a mask 38 (aperture member), a mirror 39, on the light emission side of the optical fiber 11. It includes a mask 40 (opening member) and a polarization separating element 41. Further, the illumination optical system 404 according to the eighth embodiment includes the dichroic mirror 16, the relay optical system 17, the scanning unit 18, the lens 19, the lens 20, and the objective lens 21 as in the first embodiment. The optical axis 21a of the objective lens 21 coincides with the optical axis 404a of the illumination optical system 404.

The excitation light L1 emitted from the optical fiber 11 is converted into substantially parallel light by the collimator lens 12 and enters the λ/2 wavelength plate 35. The excitation light L1 that has passed through the λ / 2 wave plate 35 includes the excitation light L1c that is linearly polarized light in the first direction and the excitation light L1d that is linearly polarized light in the second direction. The λ/2 wave plate 35 has its optical axis (fast axis, slow axis) direction set such that the light quantity of the pump light L1c and the light quantity of the pump light L1d have a predetermined ratio.

The excitation light L1 (excitation light L1c and excitation light L1d) that has passed through the λ / 2 wave plate 35 is incident on the polarization separation element 36. The polarization separation element 36 has a polarization separation film 36a tilted with respect to the optical axis 12a of the collimator lens 12. The polarization separation film 36a has a characteristic that linearly polarized light in the first direction is reflected and linearly polarized light in the second direction is transmitted. The polarization separating element 36 is, for example, a polarization beam splitter prism (PBS prism). The linearly polarized light in the first direction is S polarized light with respect to the polarization separation film 36a. The linearly polarized light in the second direction is P-polarized light with respect to the polarization separation film 36a.

The excitation light L1c, which is S-polarized light with respect to the polarization separation membrane 36a, is reflected by the polarization separation membrane 36a and is incident on the mask 38 via the mirror 37. The P-polarized excitation light L1d for the polarization separation film 36a passes through the polarization separation film 36a and enters the mask 40 via the mirror 39. The mask 38 and the mask 40 are luminous flux dividing portions that divide the excitation light that excites the fluorescent substance into a plurality of luminous fluxes. The mask 38 and the mask 40 will be described later with reference to FIG.

The excitation light L1c that has passed through the mask 38 and the excitation light L1d that has passed through the mask 40 respectively enter the polarization separation element 41. The polarization separation element 41 has a polarization separation membrane 41a inclined with respect to the optical path of the excitation light L1c and the optical path of the excitation light L1d. The polarization separation film 41a has a characteristic that linearly polarized light in the first direction is reflected and linearly polarized light in the second direction is transmitted. The polarization separating element 41 is, for example, a polarization beam splitter prism (PBS prism). The linearly polarized light in the first direction is S polarized light with respect to the polarization separation film 41a. The linearly polarized light in the second direction is P polarized light with respect to the polarization separation film 41a.

The excitation light L1c is S-polarized with respect to the polarization separation film 41a, is reflected by the polarization separation film 41a, and enters the dichroic mirror 16. The excitation light L1d is P-polarized with respect to the polarization separation film 41a, passes through the polarization separation film 41a, and enters the dichroic mirror 16. Note that one or both of the polarization separation element 36 and the polarization separation element 41 may not be the PBS prism. One or both of the polarization separating element 36 and the polarization separating element 41 may be a photonic crystal having different reflection and transmission between TE polarized light and TM polarized light.

FIG. 18 is a diagram showing the mask and the polarization state of the excitation light according to the eighth embodiment. In FIG. 18A, the Xc direction, the Yc direction, and the Zc direction are the directions corresponding to the X direction, the Y direction, and the Z direction on the sample surface Sa (see FIG. 17), respectively. The mask 38 has an opening 38a and an opening 38b. The openings 38a and 38b are arranged in the Xc direction. The openings 38a and 38b are, for example, circular, but may have shapes other than circular. The mask 38 is arranged at or near the position of a pupil conjugate plane optically conjugate with the pupil plane P0 of the objective lens 21.

In FIG. 18B, the Xd direction, the Yd direction, and the Zd direction are the directions corresponding to the X direction, the Y direction, and the Z direction on the sample surface Sa (see FIG. 17), respectively. The mask 40 has an opening 40a and an opening 40b. The openings 40a and 40b are arranged in the Yd direction. The openings 40a and 40b are, for example, circular, but may have shapes other than circular. The mask 40 is arranged at or near the position of the pupil conjugate surface that is optically conjugate with the pupil surface P0 of the objective lens 21. In the mask 38 and the mask 40, the vicinity of the pupil conjugate plane optically conjugate with the pupil plane P0 of the objective lens 21 is a range in which the excitation light L1 can be regarded as a parallel light ray in a region including the pupil conjugate plane. For example, when the excitation light L1 is a Gaussian beam, it can be sufficiently regarded as a parallel light ray within a range within 1/10 of the Rayleigh length from the beam waist position. Rayleigh length of the wavelength of the excitation light L1 lambda, when the beam waist radius was w _0, is given by πw 0 ₂ ^/ λ. For example, when the wavelength of the excitation light L1 is 1 μm and the beam waist radius is 1 mm, the Rayleigh length is about 3 m, and the mask 38 or the mask 40 is near the pupil conjugate plane optically conjugate with the pupil plane P0 of the objective lens 21. It may be arranged within 300 mm. The mask 38 or the mask 40 may be arranged at or near the pupil plane P0.

In FIG. 18C, the reference numeral AR2a is a region where the excitation light L1c passing through the opening 38a of the mask 38 is incident on the pupil surface P0 of the objective lens 21. Reference numeral AR2b is a region on the pupil surface P0 where the excitation light L1c passing through the opening 38b of the mask 38 is incident. The arrows in the regions AR2a and AR2b indicate the polarization directions of the incident excitation light L1c. The area AR2a and the area AR2b are arranged in the X direction.

The excitation light L1c incident on the region AR2a and the excitation light L1c incident on the region AR2b are linearly polarized light in the Y direction, respectively. The excitation light L1c incident on the region AR2a and the excitation light L1c incident on the region AR2b have the same polarization direction and interfere with each other on the sample surface Sa (see FIG. 17). Due to this interference, interference fringes whose periodic direction is the X direction are formed on the sample surface Sa. The incident surface of the excitation light L1c with respect to the sample surface Sa is the XZ surface, and the excitation light L1c is incident on the sample S with S polarization.

Further, in FIG. 18C, reference numeral AR2c is a region on the pupil plane P0 on which the excitation light L1d passing through the opening 40a of the mask 40 is incident. Reference numeral AR2d is a region on the pupil plane P0 on which the excitation light L1d passing through the opening 40b of the mask 40 is incident. The arrows in the regions AR2c and AR2d indicate the polarization direction of the incident excitation light L1d. The area AR2c and the area AR2d are arranged in the Y direction.

The excitation light L1d incident on the area AR2c and the excitation light L1d incident on the area AR2d are each linearly polarized in the X direction. The excitation light L1d entering the area AR2c and the excitation light L1d entering the area AR2d have the same polarization direction and interfere with each other on the sample surface Sa (see FIG. 18). Due to this interference, interference fringes whose periodic direction is the Y direction are formed on the sample surface Sa. The incident surface of the excitation light L1d on the sample surface Sa is the YZ plane, and the excitation light L1c is incident on the sample S as S-polarized light.

Returning to the explanation of FIG. 17, on the sample surface Sa, an interference fringe L2 is formed by synthesizing an interference fringe due to the interference of the excitation light L1c and an interference fringe due to the interference of the excitation light L1d. Since the polarization directions of the excitation light L1c and the excitation light L1d are substantially orthogonal to each other, interference between the excitation light L1c and the excitation light L1d is suppressed.

The detection device 6 detects the fluorescence L3 from the sample S via the detection optical system 5. As described in the first embodiment, the detection device 6 is an image sensor in which a plurality of detection units 6a are arranged in two directions, an Xb direction and a Yb direction. The image processing unit 7 generates an image (for example, a super-resolution image) by any of the processes described in the first to fourth embodiments based on the detection result of the detection device 6.

In the eighth embodiment, reflecting the fact that the illumination pupil is divided into four poles (four regions) on the pupil plane, interference fringes in the X direction and interference fringes in the Y direction simultaneously occur on the sample surface Sa. .. When the image processing phase shift processing described in the third embodiment is applied, the five components of the 0th-order component, the ± 1st-order component, and the ± 2nd-order component are used in the above equation (13) and the like. In the embodiment, the 0th-order component, the + 1st-order component in the X direction, the -1st-order component in the X direction, the + 2nd-order component in the X direction, the -2nd-order component in the X direction, the + 1st-order component in the Y direction, and -1 in the Y direction. Nine components of the second component, the +secondary component in the Y direction, and the −secondary component in the Y direction may be used. When the deconvolution in the frequency space described in the fourth embodiment is applied, the above equation (28) and the like use five components of the 0th-order component, the ± 1st-order component, and the ± 2nd-order component. In the present embodiment, the 0th-order component, the + 1st-order component in the X direction, the -1st-order component in the X direction, the +2nd-order component in the X direction, the -2nd-order component in the X direction, the + 1st-order component in the Y direction, and-in the Y direction. It is only necessary to use nine components, which are the primary component, the +secondary component in the Y direction, and the −secondary component in the Y direction. The high-order components in the XY directions may be used not only in the case of two-photon excitation but also in the case of multiphoton excitation.

According to the eighth embodiment, the effective PSFs of the images obtained by the detection units 6a can be aligned by any of the processes described in the first to fourth embodiments. Therefore, it is possible to generate an image (for example, a super-resolution image) with high resolution while ensuring S/N. The fluorescent substance contained in the sample S is multiphoton excited by the excitation light L1. As a result, similar to the first to fourth embodiments, the image data on the sample surface Sa set in an arbitrary portion including the inside of the sample S can be acquired with high accuracy.

[Ninth Embodiment]
The ninth embodiment will be described. In the present embodiment, the same components as those in the above-described embodiment will be designated by the same reference numerals, and the description thereof will be omitted or simplified. FIG. 19 is a diagram showing a microscope 501 according to the ninth embodiment. The microscope 501 according to the ninth embodiment has the same configuration as the microscope 401 according to the eighth embodiment. The microscope 501 according to the ninth embodiment includes the λ/2 wave plate 30 and the optical path rotating unit 31 described in FIG. 13. The optical path rotating unit 31 is driven by the driving unit 32 and rotates around the optical axis of the illumination optical system 404. When the optical path rotating unit 31 rotates, the optical path of the excitation light L1c and the optical path of the excitation light L1d respectively rotate around the optical axis of the illumination optical system 404. As a result, the periodic direction of the interference fringe L2 formed on the sample surface Sa rotates around the Z direction.

FIG. 20 is a diagram showing the polarization state of the excitation light according to the ninth embodiment. In FIG. 20A, the regions AR4a on which the excitation light L1c is incident on the pupil surface P0 are aligned in the X direction. Further, the area AR4b and the area AR4b on which the excitation light L1d is incident on the pupil plane P0 are arranged in the Y direction.

FIG. 20(B) corresponds to a state in which the Dove prism (optical path rotating unit 31 in FIG. 19) and the λ/2 wave plate 30 are rotated by 22.5° from the state in FIG. 20(A). In FIG. 20B, the regions AR4a on which the excitation light L1c is incident on the pupil surface P0 are arranged in a direction rotated by 45 ° from the X direction. In this state, the periodic direction of the interference fringes of the excitation light L1c on the sample surface Sa is a direction rotated by 45° from the X direction. Further, the regions AR4b on the pupil plane P0 on which the excitation light L1d is incident are arranged in the direction rotated by 45° from the Y direction. In this state, the periodic direction of the interference fringes of the excitation light L1d on the sample surface Sa is a direction rotated by 45° from the Y direction.

Returning to the description of FIG. 19, in the present embodiment, the detection device 6 detects the fluorescence L3 from the sample S before and after the cycle direction of the interference fringe L2 is changed. The image processing unit 7 starts from the first embodiment based on the detection result of the detection device 6 before the change of the periodic direction of the interference fringes L2 and the detection result of the detection device 6 after the change of the periodic direction of the interference fringes L2. An image (eg, super-resolution image) is generated by any of the processes described in the fourth embodiment. The optical path rotating unit 31 may be arranged in the optical path between the dichroic mirror 16 and the detection device 6 as described with reference to FIG.

According to the ninth embodiment, the effective PSF of the image obtained for each detection unit 6a can be made uniform by any of the processes described in the first to fourth embodiments. Therefore, it is possible to generate an image (for example, a super-resolution image) with high resolution while ensuring S/N. The fluorescent substance contained in the sample S is multiphoton excited by the excitation light L1. As a result, similar to the first to fourth embodiments, the image data on the sample surface Sa set in an arbitrary portion including the inside of the sample S can be acquired with high accuracy.

[Tenth Embodiment]
The tenth embodiment will be described. In the present embodiment, the same components as those in the above-described embodiment will be designated by the same reference numerals, and the description thereof will be omitted or simplified. The microscope 501 according to the ninth embodiment changes the periodic direction of the interference fringes L2 by the optical path rotating unit 31, but the fringe direction changing unit that changes the periodic direction of the interference fringes L2 is also different from the optical path rotating unit 31. Good.

FIG. 21 is a diagram showing a microscope 601 according to the tenth embodiment. FIG. 22 is a diagram showing a mask according to the tenth embodiment. The microscope 601 according to the tenth embodiment has the same configuration as the microscope 401 according to the eighth embodiment. The microscope 601 according to the tenth embodiment includes a drive unit 45 and a drive unit 46. The mask 38 is rotatable around the optical axis of the illumination optical system 404. The mask 38 is driven by the drive unit 45 to rotate (see FIG. 22A). In FIG. 22(A), the mask 38 is rotated clockwise by 45°.

Also, the mask 40 is rotatable around the optical axis of the illumination optical system 404. The mask 40 is driven by the drive unit 46 to rotate (see FIG. 22B). The drive unit 46 rotates the mask 40 by the same angle as the drive unit 45 rotates the mask 38. In FIG. 22B, the mask 40 is rotated clockwise by 45°. As a result, the periodic direction of the interference fringes L2 on the sample surface Sa rotates 45° around the Z direction.

A λ/2 wavelength plate 48 is provided in the optical path between the polarization separation element 41 and the dichroic mirror 16. The λ/2 wave plate 48 is driven by the drive unit 49 and rotates around the optical axis of the illumination optical system 404. The λ/2 wave plate 48 and the drive unit 49 adjust the excitation light L1c and the excitation light L1d so that they are incident on the sample S as S-polarized light.

In the present embodiment, the detection device 6 detects the fluorescence L3 from the sample S before and after the period direction of the interference fringes L2 is changed, as in the ninth embodiment. The image processing unit 7 starts from the first embodiment based on the detection result of the detection device 6 before the change of the periodic direction of the interference fringe L2 and the detection result of the detection device 6 after the change of the periodic direction of the interference fringe L2. An image (eg, super-resolution image) is generated by any of the processes described in the fourth embodiment.

According to the tenth embodiment, the effective PSF of the image obtained for each detection unit 6a can be made uniform by any of the processes described in the first to fourth embodiments. Therefore, it is possible to generate an image (for example, a super-resolution image) with high resolution while ensuring S/N. The fluorescent substance contained in the sample S is multiphoton excited by the excitation light L1. As a result, similar to the first to fourth embodiments, the image data on the sample surface Sa set in an arbitrary portion including the inside of the sample S can be acquired with high accuracy.

[11th Embodiment]
The eleventh embodiment will be described. In the present embodiment, the same components as those in the above-described embodiment will be designated by the same reference numerals, and the description thereof will be omitted or simplified. FIG. 23 is a diagram showing a microscope 701 according to the eleventh embodiment. The microscope 701 according to the eleventh embodiment has the same configuration as the microscope 1 according to the first embodiment. The microscope 701 according to the eleventh embodiment includes a relay optical system 47. The relay optical system 47 is a part of the illumination optical system 4 and a part of the detection optical system 5. The relay optical system 47 is arranged in the scanning unit 18 in the optical path between the deflection mirror 18a and the deflection mirror 18b. The deflection mirror 18b is arranged at substantially the same position as the first pupil conjugate plane optically conjugate with the pupil plane P0 of the objective lens 21. The relay optical system 47 is provided so that a second pupil conjugate surface that is optically conjugate with the first pupil conjugate surface is formed between the relay optical system 47 and the relay optical system 17. The deflection mirror 18a is arranged at substantially the same position as the second pupil conjugate plane described above.

In the present embodiment, the detection device 6 detects the fluorescence L3 from the sample S before and after the period direction of the interference fringe L2 is changed, as in the first embodiment. The image processing unit 7 starts from the first embodiment based on the detection result of the detection device 6 before the change of the periodic direction of the interference fringes L2 and the detection result of the detection device 6 after the change of the periodic direction of the interference fringes L2. An image (eg, super-resolution image) is generated by any of the processes described in the fourth embodiment. In the present embodiment, the configuration in which the relay optical system 47 is provided in the microscope 1 according to the first embodiment is illustrated, but the present invention is not limited to this, and the relay optical system 47 is provided in the microscopes according to the second to tenth embodiments. May be provided.

According to the eleventh embodiment, the effective PSF of the image obtained for each detection unit 6a can be made uniform by any of the processes described in the first to fourth embodiments. Therefore, it is possible to generate an image (for example, a super-resolution image) with high resolution while ensuring S/N. The fluorescent substance contained in the sample S is multiphoton excited by the excitation light L1. As a result, similar to the first to fourth embodiments, the image data on the sample surface Sa set in an arbitrary portion including the inside of the sample S can be acquired with high accuracy.

In each of the above-described embodiments, the scanning unit 18 is not limited to the above-described form. For example, the stage 2 may include a Y stage that moves in the Y direction with respect to the objective lens 21, and the scanning unit 18 may include a Y stage instead of the deflection mirror 18b. In this case, the scanning unit 18 may scan the sample S with the excitation light L1 in the X direction by the deflection mirror 18a, and may scan the sample S with the excitation light L1 in the Y direction by moving the Y stage. In this case, the deflection mirror 18a may be arranged at substantially the same position as the pupil conjugate plane optically conjugate with the pupil plane P0 of the objective lens 21.

Further, the stage 2 may include an X stage that moves in the X direction with respect to the objective lens 21, and the scanning unit 18 may include an X stage instead of the deflection mirror 18a. In this case, the scanning unit 18 may scan the sample S with the excitation light L1 in the X direction by moving the X stage, and may scan the sample S with the excitation light L1 in the Y direction by the deflection mirror 18b. In this case, the deflection mirror 18b may be arranged at substantially the same position as the pupil conjugate plane optically conjugate with the pupil plane P0 of the objective lens 21.

Further, the stage 2 includes an X stage that moves in the X direction with respect to the objective lens 21 and a Y stage that moves in the Y direction with respect to the objective lens 21, and the scanning unit 18 includes the X stage and the Y stage described above. May be included. In this case, the scanning unit 18 may scan the sample S with the excitation light L1 in the X direction by moving the X stage, and may scan the sample S with the excitation light L1 in the Y direction by moving the Y stage. ..

In each of the above-described embodiments, the scanning direction for scanning the sample S with the interference fringes is two directions, the X direction and the Y direction, and the illumination optical system scans the sample S two-dimensionally with the interference fringes. The scanning direction in which the sample S is scanned with the interference fringes may be the X direction, the Y direction, and the Z direction. For example, the microscope according to each embodiment performs a 2D process of scanning the sample S with interference fringes in the X and Y directions to acquire a 2D image, for example, moving at least one of the objective lens 21 and the stage 2. The sample S may be three-dimensionally scanned with the interference fringes by changing the Z-direction position where the interference fringes are generated and repeating the 2D processing. The microscope according to each embodiment acquires a plurality of 2D images having different positions in the Z direction by three-dimensionally scanning the sample S with interference fringes, and generates a 3D image (eg, Z-stack). Good. When the sample S is three-dimensionally scanned with interference fringes, the illumination optical system 4 may scan in the X direction and the Y direction, and may scan in the Z direction by moving at least one of the objective lens 21 and the stage 2. The illumination optical system 4 may scan the sample S three-dimensionally with interference fringes.

In each of the above embodiments, the image processing unit 7 includes, for example, a computer system. The image processing unit 7 reads out the image processing program stored in the storage unit and executes various processes according to the image processing program. This image processing program causes a computer to generate an image based on the detection results of the detection devices 6 and 106. The detection results of the detection devices 6 and 106 are obtained by dividing the light from the light source into a plurality of light beams, and scanning the sample in a plurality of directions by the interference fringes generated by the interference of at least a part of the light beams. Then, the light from the sample is detected by the detection device including the plurality of detection units via the detection optical system on which the light from the sample is incident.

[Modification]
Hereinafter, a modified example will be described. The same reference numerals are given to the same configurations as those of the above-described embodiments, and the description thereof will be omitted or simplified. 24 and 25 are diagrams showing an illumination pupil according to a modified example.

The illumination pupil has two poles in FIG. 2, four poles in FIG. 18, but three poles in FIG. 24 (A). Reference numerals AR5a to AR5c are regions on the pupil plane P0 on which the excitation light enters. In this case, the first interference fringe between the excitation light incident on the region AR5a and the excitation light incident on the region AR5b, the second interference fringe between the excitation light incident on the region AR5b and the excitation light incident on the region AR5c, and the region. Third interference fringes of the excitation light incident on the AR5c and the excitation light incident on the area AR5a are formed. On the sample surface Sa, interference fringes that are a combination of the first interference fringes, the second interference fringes, and the third interference fringes are formed. In the interference fringes, the periodic direction of the first interference fringes, the periodic direction of the second interference fringes, and the periodic direction of the third interference fringes are the respective periodic directions, and the periodic directions are three directions. A super-resolution effect can be obtained. Note that the illumination pupil may have 5 or more poles.

Also, the illumination pupil is circular in FIG. 2 and the like, but may have other shapes. In FIG. 24(B) and FIG. 24(C), the code|symbol AR6 is an area|region which excitation light injects. The area AR6 in FIG. 24B is an area surrounded by a circle AR6a which is a part of a circle centered on the optical axis 21a of the objective lens 21 and a straight line AR6b connecting both ends of the arc AR6a. The area AR6 in FIG. 24C is an area surrounded by an arc that is a part of a circle centered on the optical axis 21a of the objective lens 21 and a curved line AR6c that is symmetrical to the arc AR6a.

In the case of the illumination pupil having the shape of FIG. 24(B) or FIG. 24(C), the resolution in the direction in which the interference fringes are not formed is better and the sectioning is better than that in the case of the circular illumination pupil. Further, in the case of the illumination pupil having the shape of FIG. 24(B), the resolution in the direction in which the interference fringes are not formed is better and the sectioning is also better than that in the case of the illumination pupil of the shape of FIG. 24(C). .. Further, in the case of the illumination pupil having the shape of FIG. 24C, the resolution in the direction in which the interference fringes are formed is better than that in the case of the illumination pupil having the shape of FIG.

In FIG. 25 (A), the illuminated pupil has a shape in which the illuminated pupil having the shape shown in FIG. 24 (B) has four poles. In FIG. 25 (B), the illuminated pupil has a shape in which the illuminated pupil having the shape shown in FIG. 24 (C) has four poles. Even in the case of the illumination pupil having a shape other than the circular shape, the number of the plurality of regions (the number of poles) on which the excitation light enters is set to an arbitrary number of 2 or more. In addition, the shape of one of the plurality of areas on the pupil plane P0 on which the excitation light is incident may be different from the shape of the other areas. Further, of the plurality of regions on the pupil plane P0 where the excitation light is incident, the size of one region may be different from the size of the other regions. Further, the plurality of regions in which the excitation light is incident on the pupil surface P0 may be arranged asymmetrically with respect to the optical axis 21a of the objective lens 21.

The shape, dimensions, and arrangement of each pole of the illumination pupil can be realized, for example, by designing the shape, dimensions, and arrangement of the opening of the mask 15 shown in FIG. Further, the mask 15 may be a mechanical diaphragm having a variable light blocking area, a spatial light modulator (SLM), or the like.

FIG. 26 is a diagram showing a microscope 711 according to the first modification. Although FIG. 26 shows a modification applied to the microscope 1 according to the first embodiment as a modification of the microscope, it can also be applied to microscopes of other embodiments. In FIG. 26, the illumination optical system 4 includes a collimator lens 50, a λ/2 wavelength plate 51, a lens 52, a diffraction grating 53, a lens 54, and a mask 15 in the order from the optical fiber 11 to the dichroic mirror 16. The collimator lens 50 converts the excitation light L1 from the optical fiber 11 into substantially parallel light. The λ/2 wave plate 51 adjusts the polarization state of the excitation light L1 when entering the sample S. The lens 52 concentrates the excitation light L1 on the diffraction grating 53.

The diffraction grating 53 splits the excitation light L1 into a plurality of light beams by diffraction. The diffraction grating 53 is a light beam splitting unit that splits the excitation light that multi-photon excites the fluorescent material into a plurality of light beams. The diffraction grating 53 is arranged at the focal point or a position near the focal point of the lens 52. That is, the diffraction grating 53 is arranged on the surface conjugate with the sample surface Sa or in the vicinity thereof. The vicinity of the focus is within the range of the depth of focus of the light condensed by the lens 52. For example, when the wavelength of light condensed by the lens 52 is 1 μm and the NA is 0.03, the depth of focus is about 1 mm, so the diffraction grating 53 may be arranged within 1 mm near the focus of the lens 52. The plurality of light fluxes include 0th-order diffracted light, +1st-order diffracted light, and −1st-order diffracted light. The lens 54 converts the 0th-order diffracted light, the +1st-order diffracted light, and the -1st-order diffracted light into substantially parallel lights, respectively. The mask 15 is provided so as to block the 0th-order diffracted light and pass at least a part of the +1st-order diffracted light and at least a part of the −1st-order diffracted light. In such a form, the light amount of the excitation light L1 that passes through the mask 15 can be increased. The diffraction grating 53 may be designed so that the 0th-order diffracted light is not generated. Further, a configuration may be adopted in which the mask 15 is not provided. Further, the diffraction grating 53 may be designed so that diffracted light is generated in a plurality of directions. In that case, the mask 15 may be rotationally driven to use only the diffracted light in a specific direction.

27 and 28 are diagrams showing a polarization adjusting unit according to a modification, respectively. Although the optical path of the illumination optical system 4 is bent by the reflecting member such as the dichroic mirror 16 shown in FIG. 1, in FIGS. 27 and 28, the illumination optical system 4 is expanded so that the optical axis 4a becomes a straight line. Indicated. In FIGS. 27 and 28, the Z direction is a direction parallel to the optical axis 4a, and the X direction and the Y direction are directions perpendicular to the optical axis 4a, respectively.

In FIG. 27, the illumination optical system 4 includes a λ/4 wavelength plate 61, a mask 15, and a λ/4 wavelength plate 62. The excitation light L1 emitted from the optical fiber 11 is linearly polarized in the X direction and is incident on the λ/4 wavelength plate 61. A polarizer (eg, a polarizing plate) having a transmission axis in the X direction may be provided in the optical path between the optical fiber 11 and the λ/4 wavelength plate 61.

The phase-advancing axis of the λ / 4 wave plate 61 is set in the direction in which the X direction is rotated counterclockwise by 45 ° when viewed from the + Z side. The excitation light L1 that has passed through the λ/4 wavelength plate 61 becomes circularly polarized light and enters the mask 15. The excitation light L1 that has passed through the openings 15a and 15b of the mask 15 is circularly polarized light and is incident on the λ/4 wavelength plate 62. The fast axis of the λ/4 wave plate 62 is set in a direction obtained by rotating the X direction clockwise by 45° when viewed from the +Z side. The excitation light L1 that has passed through the λ/4 wavelength plate 62 becomes linearly polarized light in the X direction and is applied to the sample surface Sa.

The mask 15 is rotatably provided around the optical axis 4a as described in the first embodiment. When the mask 15 rotates, the periodic direction of the interference fringes changes. For example, in the state of FIG. 27, the openings 15a and 15b of the mask 15 are aligned in the Y direction, and the periodic direction of the interference fringes is the Y direction. When the mask 15 is rotated 90° from the state shown in FIG. 27, the periodic direction of the interference fringes is rotated 90° and becomes the X direction.

The λ/4 wave plate 62 is rotatable around the optical axis 4a. The λ/4 wave plate 62 is provided so as to rotate by the same angle as the mask 15. For example, the λ/4 wave plate 62 is integrated with the mask 15 and rotates integrally with the mask 15. For example, when the mask 15 is rotated by 90 °, the λ / 4 wave plate 62 is rotated by 90 ° and its phase advance axis becomes parallel to the phase advance axis of the λ / 4 wave plate 61. In this case, the excitation light L1 that has passed through the λ/4 wavelength plate 62 becomes linearly polarized light in the Y direction. The incident surface of the excitation light L1 with respect to the sample surface Sa is parallel to the periodic direction of the interference fringes, and the excitation light L1 when incident on the sample surface Sa is linearly polarized light perpendicular to the periodic direction of the interference fringes. The sample surface Sa is irradiated with L1 in the S-polarized state.

As described above, the λ / 4 wave plate 62 is included in the polarization adjusting unit that adjusts the polarization state of the excitation light L1 when it is incident on the sample S. Such a polarization adjusting unit can reduce the loss of the light amount of the excitation light L1 as compared with the aspect described in FIG.

In FIG. 28, the illumination optical system 4 includes a polarizer 65, a mask 15, and a λ/2 wave plate 66. The excitation light L1 emitted from the optical fiber 11 is linearly polarized light in the X direction and is incident on the polarizer 65. The transmission axis of the polarizer 65 is set in the X direction. The excitation light L1 that has passed through the polarizer 65 is linearly polarized in the X direction and is incident on the mask 15. The excitation light L1 that has passed through the openings 15a and 15b of the mask 15 is linearly polarized light in the X direction and is incident on the λ/2 wave plate 66. The fast axis of the λ/2 wave plate 66 is set in a direction obtained by rotating the X direction clockwise by 45° when viewed from the +Z side. The excitation light L1 that has passed through the λ/2 wavelength plate 66 becomes linearly polarized light in the Y direction and is applied to the sample surface Sa.

The mask 15 is rotatably provided around the optical axis 4a as described in the first embodiment. When the mask 15 rotates, the periodic direction of the interference fringes changes. For example, in the state of FIG. 28, the openings 15a and 15b of the mask 15 are aligned in the X direction, and the periodic direction of the interference fringes is the X direction. When the mask 15 is rotated by 90° from the state of FIG. 28, the periodic direction of the interference fringes is rotated by 90° and becomes the Y direction.

The λ/2 wave plate 66 is rotatable around the optical axis 4a. The λ/2 wave plate 66 is provided so as to rotate by an angle half the rotation angle of the mask 15. For example, when the mask 15 rotates 90°, the λ/2 wave plate 66 rotates 45°. In this case, the excitation light L1 that has passed through the λ/2 wave plate 66 becomes linearly polarized light in the X direction. The incident surface of the excitation light L1 with respect to the sample surface Sa is parallel to the periodic direction of the interference fringes, and the excitation light L1 when incident on the sample surface Sa is linearly polarized light perpendicular to the periodic direction of the interference fringes. The sample surface Sa is irradiated with L1 in the S-polarized state. In this way, the λ/2 wave plate 66 is included in the polarization adjusting unit that adjusts the polarization state of the excitation light when entering the sample S. Such a polarization adjusting unit can reduce the loss of the light amount of the excitation light L1 as compared with the aspect described in FIG. Although the polarization adjusting unit described in the above-described embodiment and modification is configured such that the illumination light becomes linearly polarized light immediately after passing through the polarization adjusting unit, it may not be perfect linearly polarized light. Further, an additional polarizing element such as a λ/2 wavelength plate or a λ/4 wavelength plate may be added to the polarization adjusting unit to correct the change in the polarization state generated in the optical system on the way.

When the microscope according to each embodiment includes the detection device 6 including an image sensor, the microscope may include an image rotation unit that rotates the image of the sample S around the optical axis of the detection optical system 5. When the periodic direction of the interference fringes is rotated, the periodic direction of the interference fringes and the arrangement direction of the detection unit 6a can be matched by rotating the image of the sample S.

The microscope according to each embodiment may include a phase modulation unit that can shift the phase of the interference fringe L2 formed by the excitation light L1 to set the interference fringe L2 in a plurality of phase states. .. Hereinafter, shifting the phase of the interference fringe L2 formed by the excitation light L1 may be referred to as changing the phase. FIG. 29 is a diagram showing a microscope 721 according to the second modification.

The microscope 721 according to the second modification has the same configuration as the microscope 711 according to the first modification. The microscope 721 according to the second modified example includes a phase modulator 55. The phase modulation unit 55 includes a drive unit capable of moving the diffraction grating 53 in a direction perpendicular to the optical axis 4a of the illumination optical system 4. The phase modulation unit 55 shifts the phase of the fringes of the interference fringes L2 by moving the diffraction grating 53 in parallel in the direction perpendicular to the optical axis 4a of the illumination optical system 4 (the direction corresponding to the periodic direction of the interference fringes L2). Can be made.

In another example of the phase modulation unit, in the vicinity of the pupil conjugate surface that is optically conjugated with the pupil surface P0 of the objective lens 21, an optical path having an optical path length different for each light beam divided by the light beam dividing portion is passed through. , The phase of the interference fringe L2 is changed. FIG. 30 is a diagram showing a microscope 731 according to the third modification. Although FIG. 30 shows a modification applied to the microscope 1 according to the first embodiment as a modification of the microscope, it can also be applied to microscopes of other embodiments.

The microscope 731 according to the third modification has the same configuration as the microscope 1 according to the first embodiment. A microscope 731 according to the third modified example includes a phase modulator 70. The phase modulator 70 includes a phase modulator 71 and a driver 78. The phase modulation element 71 is arranged between the mask 15 and the dichroic mirror 16. The phase modulation element 71 is formed into a circular plate shape by performing removal processing such as cutting on a predetermined glass plate. The phase modulation element 71 is arranged so that the central axis of the phase modulation element 71 coincides with the optical axis 4 a of the illumination optical system 4. The drive unit 78 rotates the phase modulation element 71 around the optical axis of the illumination optical system 4. The phase modulation element 71 may be arranged in a portion of the optical path of the parallel light flux in the illumination optical system 4 that does not overlap with the optical path of the detection optical system 5.

FIG. 31 is a diagram showing details of the phase modulation element 71. The phase modulation element 71 has a first plate part 72, a second plate part 73, a third plate part 74, a fourth plate part 75, and a fifth plate part 76 having different thicknesses. The first plate portion 72 has the thickest first thickness TH1 among the first to fifth plate portions 72 to 76. The first plate portion 72 has, for example, a fan shape with a central angle of 216° when the phase modulation element 71 is viewed in the thickness direction (optical axis direction).

The second plate portion 73 has the second thickest second thickness TH2 of the first to fifth plate portions 72 to 76. The second plate portion 73 has, for example, a fan shape having a central angle of 36° when the phase modulation element 71 is viewed in the thickness direction (optical axis direction). When the phase modulation element 71 is visually recognized from the thickness direction (optical axis direction), the second plate portion 73 is in the circumferential direction of the first plate portion 72 (counterclockwise direction when viewed from the paper surface side of FIG. 31). Placed next to each other.

The third plate portion 74 has a third thickness TH3, which is the third largest among the first to fifth plate portions 72 to 76. The third plate portion 74 has, for example, a fan shape having a central angle of 36° when the phase modulation element 71 is viewed in the thickness direction (optical axis direction). When the phase modulation element 71 is visually recognized from the thickness direction (optical axis direction), the third plate portion 74 is in the circumferential direction of the second plate portion 73 (counterclockwise direction when viewed from the paper surface side of FIG. 31). Placed next to each other.

The fourth plate portion 75 has the fourth thickest fourth thickness TH4 of the first to fifth plate portions 72 to 76. The fourth plate portion 75 has, for example, a fan shape having a central angle of 36° when the phase modulation element 71 is viewed in the thickness direction (optical axis direction). When the phase modulation element 71 is visually recognized from the thickness direction (optical axis direction), the fourth plate portion 75 is in the circumferential direction of the third plate portion 74 (counterclockwise direction when viewed from the paper surface side of FIG. 31). Placed next to each other.

The fifth plate portion 76 has a fifth thickness TH5 that is the fifth largest among the first to fifth plate portions 72 to 76. The fifth plate portion 76 has, for example, a fan shape having a central angle of 36° when the phase modulation element 71 is viewed in the thickness direction (optical axis direction). The fifth plate portion 76 is arranged in the circumferential direction of the fourth plate portion 75 (counterclockwise when viewed from the paper surface side of FIG. 31) when the phase modulation element 71 is viewed in the thickness direction (optical axis direction). Placed next to each other.

One of the optical surfaces of the phase modulation element 71 is formed into a flat surface having a step due to the first to fifth plate portions 72 to 76. The other optical surface of the phase modulation element 71 is formed in a flat plane shape as a whole. In the phase modulation element 71, for example, as shown in FIG. 30, an optical surface having steps formed by the first to fifth plate portions 72 to 76 is arranged so as to face the mask 15. The phase modulation element 71 may be arranged such that the optical surface having the steps formed by the first to fifth plate portions 72 to 76 faces the dichroic mirror 16.

32(A) to 32(E) are diagrams showing the positional relationship between the phase modulation element 71 and the excitation light L1. As described above, the first plate portion 72 has a fan-shaped shape with a central angle of 216° which is larger than 180° when the phase modulation element 71 is viewed in the thickness direction (optical axis direction). .. Therefore, as shown in FIG. 32(A), when the phase modulation element 71 is rotationally moved to the predetermined first rotational position by the rotational driving force of the driving unit 78, the excitation light L1a and the mask 15 that have passed through the opening 15a of the mask 15. Both of the excitation lights L1b that have passed through the opening 15b are transmitted through the first plate portion 72. At this time, there is no difference between the optical path length of the pumping light L1a and the optical path length of the pumping light L1b.

As shown in FIG. 32(B), when the phase modulation element 71 is rotationally moved to the predetermined second rotational position by the rotational driving force of the driving unit 78, the excitation light L1a passing through the opening 15a of the mask 15 becomes the first plate portion. The excitation light L1b that passes through 72 and passes through the opening 15b of the mask 15 passes through the second plate portion 73. The second rotation position is, for example, when the phase modulation element 71 is rotated by 36° around the optical axis of the illumination optical system 4 (clockwise when viewed from the paper surface side of FIG. 32) with respect to the first rotation position. The rotation position. At this time, since the first plate portion 72 has the first thickness TH1 and the second plate portion 73 has the second thickness TH2 thinner than the first thickness TH1, the optical path length of the excitation light L1a. And the optical path length of the excitation light L1b is different. Therefore, the phase of the fringes of the interference fringes obtained when the phase modulation element 71 is rotationally moved to the second rotation position changes with respect to the interference fringes obtained when the phase modulation element 71 is rotationally moved to the first rotation position. To do.

As shown in FIG. 32 (C), when the phase modulation element 71 is rotationally moved to a predetermined third rotation position by the rotational driving force of the drive unit 78, the excitation light L1a that has passed through the opening 15a of the mask 15 is the first plate portion. The excitation light L1b that passes through 72 and passes through the opening 15b of the mask 15 passes through the third plate portion 74. The third rotation position is, for example, when the phase modulation element 71 is rotated by 72° around the optical axis of the illumination optical system 4 (clockwise when viewed from the paper surface side of FIG. 32) with respect to the first rotation position. The rotation position. At this time, since the first plate portion 72 has the first thickness TH1 and the third plate portion 74 has the third thickness TH3 which is thinner than the first thickness TH1, the optical path length of the excitation light L1a is increased. And the optical path length of the excitation light L1b is different. Therefore, with respect to the interference fringes obtained when the phase modulation element 71 rotationally moves to the first rotation position, the phase of the interference fringes obtained when the phase modulation element 71 rotationally moves to the third rotation position changes. To do.

As shown in FIG. 32D, when the phase modulation element 71 is rotationally moved to the predetermined fourth rotational position by the rotational driving force of the drive unit 78, the excitation light L1a passing through the opening 15a of the mask 15 is converted into the first plate portion. The excitation light L1b that passes through 72 and passes through the opening 15b of the mask 15 passes through the fourth plate portion 75. The fourth rotation position is, for example, when the phase modulation element 71 is rotated by 108° around the optical axis of the illumination optical system 4 (clockwise when viewed from the paper surface side of FIG. 32) with respect to the first rotation position. The rotation position. At this time, since the first plate portion 72 has the first thickness TH1 and the fourth plate portion 75 has the fourth thickness TH4 smaller than the first thickness TH1, the optical path length of the excitation light L1a is reduced. And the optical path length of the excitation light L1b is different. Therefore, with respect to the interference fringes obtained when the phase modulation element 71 rotationally moves to the first rotation position, the phase of the interference fringes obtained when the phase modulation element 71 rotationally moves to the fourth rotation position changes. To do.

As shown in FIG. 32(E), when the phase modulation element 71 is rotationally moved to the predetermined fifth rotational position by the rotational driving force of the driving section 78, the excitation light L1a passing through the opening 15a of the mask 15 is converted into the first plate section. The excitation light L1b that has passed through 72 and passed through the opening 15b of the mask 15 passes through the fifth plate portion 76. The fifth rotation position is, for example, when the phase modulation element 71 is rotated by 144° around the optical axis of the illumination optical system 4 (clockwise when viewed from the paper surface side of FIG. 32) with respect to the first rotation position. The rotation position. At this time, since the first plate portion 72 has the first thickness TH1 and the fifth plate portion 76 has the fifth thickness TH5 which is thinner than the first thickness TH1, the optical path length of the excitation light L1a is increased. And the optical path length of the excitation light L1b is different. Therefore, with respect to the interference fringes obtained when the phase modulation element 71 rotationally moves to the first rotation position, the phase of the interference fringes obtained when the phase modulation element 71 rotationally moves to the fifth rotation position changes. To do.

As described above, the phase of the fringe of the interference fringe L2 can be changed by the phase modulation element 71. Changing the phase of the stripe is equivalent to changing 2πk ₀ r, which is the argument of the cosine function, to 2πk ₀ r+φ without changing PSF _ill (r) on the right side of Expression (2). .. Here, φ is the phase difference between the excitation light L1a and the excitation light L2b. The fact that PSF _ill (r) does not change means that the envelope (envelope) of the intensity distribution of the interference fringe L2 does not change. Therefore, the phase modulation element 71 changes the bright and dark phases of the intensity distribution of the interference fringe L2 without changing the envelope (envelope) of the intensity distribution of the interference fringe L2.

As described above, the phase of the fringe of the interference fringe L2 can be changed in five steps by rotating and moving the phase modulation element 71 to the first to fifth rotation positions, respectively. The thicknesses of the first to fifth plate portions 72 to 76 (first to fifth thicknesses TH1 to TH5) are set so that a necessary phase difference of the interference fringes L2 can be obtained. The thickness of the first to fifth plate portions 72 to 76 (first to fifth thicknesses TH1 to TH5) can be set by numerical simulation or the like.

For example, the phase difference between the interference fringes when the phase modulation element 71 is rotationally moved to the first rotation position and the interference fringes when the phase modulation element 71 is rotationally moved to the second rotation position is 2π/5. The thickness of the second plate portion 73 (second thickness TH2) can be set. The phase difference between the interference fringes when the phase modulation element 71 is rotationally moved to the first rotation position and the interference fringes when the phase modulation element 71 is rotationally moved to the third rotation position is 4π/5. It is possible to set the thickness of the third plate portion 74 (third thickness TH3). The phase difference between the interference fringes when the phase modulation element 71 is rotationally moved to the first rotation position and the interference fringes when the phase modulation element 71 is rotationally moved to the fourth rotation position is 6π/5. It is possible to set the thickness of the four plate portion 75 (fourth thickness TH4). The phase difference between the interference fringes when the phase modulation element 71 is rotationally moved to the first rotation position and the interference fringes when the phase modulation element 71 is rotationally moved to the fifth rotation position is 8π/5. It is possible to set the thickness (fifth thickness TH5) of the five plate portion 76.

Further, the phase modulation element 71 rotates and moves not only when changing the phase of the fringes of the interference fringe L2 but also when changing the cycle direction of the interference fringe L2. In this case, the relative positional relationship between the openings 15a and 15b of the mask 15 and the portions of the phase modulation element 71 (first to fifth plate portions 72 to 76) facing the openings 15a and 15b of the mask 15 becomes constant. In addition, the phase modulation element 71 rotationally moves according to the rotation of the mask 15.

The central angle of the fan shape in the first plate portion 72 is not limited to the illustrated angle (216°), but the excitation light L1a passing through the opening 15a of the mask 15 and the excitation light L1b passing through the opening 15b of the mask 15. It suffices that both are set to an angle that allows the first plate portion 72 to pass therethrough. The central angle of the fan shape in the second to fifth plate portions 73 to 76 is not limited to the exemplified angle (36°), and the excitation light L1a or the excitation light L1b is not included in the second to fifth plate portions 73 to 76. It suffices if the angle is set so that can be transmitted.

The phase modulation element 71 is not limited to the removal processing, and may be formed into a circular plate shape by, for example, molding processing, processing of laminating thin plate materials using a transparent resin material, or the like. Further, as the phase modulation element, a plurality of plate-shaped phase modulation elements having different thicknesses may be used. In this case, the drive unit of the phase modulation unit selects any one of the plurality of phase modulation elements and arranges it between the mask 15 and the dichroic mirror 16. As such a drive unit, for example, a slider or a turret that can move a plurality of phase modulation elements in a direction perpendicular to the optical axis of the illumination optical system 4 may be used.

By using the phase modulation unit described above, it is possible to perform component separation in a mode different from the component separation of the image processing described in the third embodiment. As described in the third embodiment, the image I(r,r _s ) obtained by the detection device 6 is represented by the above formula (12). In equation (12), φ indicates the initial phase of the interference fringe L2. By changing the phase of the interference fringe L2 using the phase modulator, the value of the initial phase φ in equation (12) changes, but the changed value of φ can also be called the phase difference. That is, even if the image data is acquired with a predetermined phase difference φ given, the obtained image data is given by the equation (12). The value of the predetermined phase difference φ includes 0. When image data is acquired in a state where a predetermined phase difference φ is given, four-dimensional frequency space data obtained by Fourier transform by the image processing unit 7 is given by I 1 ^to (r, k _s ; φ). ^~ (R, k _s ; φ) is expressed by the following equation (31).

Here, the 0th-order components I′ ₀ ^to (r, k _s ) of the equation (31) are defined by the following equation (32).

The +1st order components I′ ₊₁ ^to (r, k _s ) of the equation (31) are defined by the following equation (33).

-1-order component I _'-1 ^~ of formula (31) (r, _{k s)} is defined as the following equation (34).

The +second-order component I′ ₊₂ ^to (r, k _s ) of the equation (31) is defined by the following equation (35).

Minus second-order component I _'-2 ^~ of formula (31) (r, _{k s)} is defined as the following equation (36).

In the formula (32) to Formula _{(36), OTF 'det (} r, k s) _{is, PSF} det a (r + _{r s)} is obtained by Fourier transform for _{r s, OTF' det (r} , k s) = e it is a ^i2πksr OTF _{det (k s).} Further, OTF' _ill (k _s ) is the Fourier transform of PSF ² _ill (r _s ). Obj ^to (k _s ) are the Fourier transform of Obj(r _s ).

The microscopes 721 and 731 according to the second and third modifications change the phase of the interference fringe L2 from a predetermined phase by changing the phase of the interference fringe L2 using the phase modulation unit, and change the phase of the interference fringe L2 from a predetermined phase. When the phase difference with respect to the phase is the first phase difference φ ₁ , the second phase difference φ ₂ , the third phase difference φ ₃ , the fourth phase difference φ ₄ , and the fifth phase difference φ _5. Acquire each image data. I ₁ ^to (r, k _s ; φ ₁ ) corresponding to the first phase difference φ ₁ , I ₁ ^to (r, k _s ; φ ₂ ) corresponding to the _second phase difference φ ₂ , and the third phase difference φ corresponding to ^{_{3 I ~ (r, k s}} ; φ 3), fourth ^I ~ corresponding to the phase difference _{φ 4 (r, k s;} φ 4), and a fifth I corresponding to the phase difference phi ₅ ^~ (R, k _s ; φ ₅ ) are represented by the following expressions (37A) to (37E).

The equations (37A) to (37E) represent five unknowns I′ ₀ ^to (r, k _s ), I′ ₊₁ ^to (r, k _s ), I′ ₋₁ ^to (r, k _s ), I′ _+2. ^~ (R, k _s ) and I'- ₂ ^~ (r, k _s ) are simultaneous equations. The image processing unit 7 solves the simultaneous equations to obtain the acquired data I 1 ^to (r, k _s ; φ ₁ ), I ^to (r, k _s ; φ ₂ ), I ^to (r, k _s ; φ _{3 ).} ), I ^to (r, k _s ; φ ₄ ) and I ^to (r, k _s ; φ ₅ ), I′ ₀ ^to (r, k _s ), I′ ₊₁ ^to (r, k _s ), I ' _-1 ^to (r, k _s ), I' ₊₂ ^to (r, k _s ) and I'- ₂ ^to (r, k _s ) are obtained. This processing corresponds to the component separation of the image processing described in the third embodiment. The image processing unit 7 has I′ ₀ ^to (r, k _s ), I′ ₊₁ ^to (r, k _s ), I′ ₋₁ ^to (r, k _s ), I′ ₊₂ ^to (r, k _s ). , I′ ₋₂ ^to (r, k _s ) are inverse Fourier transformed to obtain I ₀ (r, r _s ), I ₊₁ (r, r _s ), I ₋₁ (r, r _s ), I ₊₂ (R,r _s ) and I ₋₂ (r,r _s ) can be obtained. Then, the image processing unit 7 performs the same processing as that in the third embodiment after the processing for calculating the image data in the real space in the third embodiment. As a result, even if the frequency space region in the component separation is not set, the process corresponding to the component separation can be performed by solving the simultaneous equations of the equations (37A) to (37E), and the position of the detection unit 6a can be performed. It is possible to correct the collapse of the shape of the effective PSF for each (detector coordinate r).

FIG. 33 is a flowchart showing an observation method using the microscope 731 according to the third modification. In step S41, the control unit 8 of the microscope 731 sets the periodic direction of the interference fringes L2. The control unit 8 sets the cycle direction of the interference fringes L2 by rotating the mask 15 by the drive unit 22 and rotating the phase modulation element 71 by the drive unit 78 according to the rotation of the mask 15. In step S42, the control unit 8 sets the phase of the fringe of the interference fringe L2. The control unit 8 sets the phase of the fringe of the interference fringe L2 by rotating the phase modulation element 71 by the drive unit 78. In step S43, the control unit 8 sets the angles of the scanning mirrors (deflection mirrors 18a and 18b). The illumination optical system 4 irradiates the excitation light as interference fringes L2 at a position on the sample determined by the angle of the scanning mirror set in step S43. In step S44, the fluorescent substance of the sample S is excited by the interference fringe L2 of the excitation light. In step S45, the detection device 6 detects the fluorescence from the sample S via the detection optical system 5 for each of the plurality of detection units 6a.

In step S46, the control unit 8 determines whether to change the angle of the scanning mirror. When it is determined that the processes of steps S43 to S45 have not been completed for a part of the scheduled observation region, the control unit 8 determines to execute the angle change of the scanning mirror in step S46 (step S46; Yes). ). When the control unit 8 determines to change the angle of the scanning mirror (step S46; Yes), the control unit 8 returns to the process of step S43, and the control unit 8 sets the angle of the scanning mirror to the next scheduled angle. Then, the processing from step S44 to step S46 is repeated.

When it is determined in step S46 that the processes in steps S43 to S45 have been completed for all the scheduled observation regions, the control unit 8 determines not to change the angle of the scanning mirror (step S46; No). .. When the control unit 8 determines not to change the angle of the scanning mirror (step S46; No), the control unit 8 determines whether to change the phase of the fringe of the interference fringe L2 in step S47. ..

When the control unit 8 determines that the processing of steps S42 to S46 has not been completed for a part of the phases of the five interference fringes planned, the control unit 8 changes the fringe phase of the interference fringe L2 in step S47. Is determined to be executed (step S47; Yes). When the control unit 8 determines to change the phase of the fringe of the interference fringe L2 (step S47; Yes), the control unit 8 returns to the process of step S42, and the control unit 8 schedules the phase of the fringe of the interference fringe L2. Set to the next phase. Then, the processing from step S43 to step S47 is repeated. In this way, the illumination optical system 4 two-dimensionally scans the sample S with the interference fringes L2 of the excitation light for the five interference fringes whose fringe phase is changed.

When it is determined in step S47 that the processes of steps S42 to S46 have been completed for all the phases of the five fringes of the planned interference fringes, the control unit 8 executes the phase change of the fringes of the interference fringes L2. It is determined not to do so (step S47; No). When the control unit 8 determines that the phase of the fringes of the interference fringes L2 is not changed (step S47; No), in step S48, the image processing unit 7 performs image processing to perform an image (eg, super-resolution image). ) Is generated.

In step S49, the control unit 8 determines whether to change the cycle direction of the interference fringe L2. When the control unit 8 determines in step S49 that the processing in steps S41 to S48 has not been completed for a part of the scheduled cycle of the interference fringes L2, the controller 8 changes the cycle of the interference fringes L2 in step S49. The determination is made (step S49; Yes). When the control unit 8 determines to execute the change of the periodic direction of the interference fringe L2 (step S49; Yes), the control unit 8 returns to the process of step S41, and the control unit 8 determines the next periodic direction of the interference fringe L2. Set in the periodic direction. Then, the processing from step S42 to step S49 is repeated.

When the control unit 8 determines in step S49 that the processes in steps S41 to S48 have been completed for all the scheduled periodic directions of the interference fringes L2, the control unit 8 determines not to change the periodic direction of the interference fringes L2. (Step S49; No). When the control unit 8 determines that the change in the periodic direction of the interference fringe L2 is not executed (step S49; No), the process ends. This makes it possible to generate an image in which the cycle direction of the interference fringe L2 is changed.

FIG. 34 is a flowchart showing a sub-flow of image processing by the image processing unit 7. In step S51, the image processing unit 7 selects at least a part of the plurality of detection units 6a of the detection device 6. In step S52, the image processing unit 7 Fourier transforms the detection results of at least a part of the plurality of detection units 6a selected in step S51. The image processing unit 7 performs a two-dimensional Fourier transform on I(r, r _s ; φ) given a predetermined phase difference φ. In step S53, the image processing unit 7 performs a process corresponding to the component separation in the third embodiment by solving the simultaneous equations represented by the above equations (37A) to (37E). In step S54, the image processing unit 7 performs an inverse Fourier transform on the separated components. In step S55, the image processing unit 7 executes the image processing phase shift processing. The image processing unit 7 corrects the positional deviation of the effective PSF in step S56. In step S57, the image processing unit 7 generates an image (for example, a super-resolution image) by adding the images obtained by correcting the positional deviation in step S56.

In step S58, the control unit 8 determines whether to change the selection of the detection unit 6a. When it is determined that the processes of steps S51 to S57 have not been completed for a part of the planned combination of the detection units 6a, the control unit 8 determines to execute the selection change of the detection unit 6a in step S58 ( Step S58; Yes). When the control unit 8 determines to change the selection of the detection unit 6a (step S58; Yes), the control unit 8 returns to the process of step S51, and the control unit 8 selects at least another part of the plurality of detection units 6a. .. Then, the processing from step S52 to step S58 is repeated.

When the control unit 8 determines in step S58 that the processing of steps S51 to S57 has been completed for all the scheduled combinations of the detection units 6a, it determines that the selection change of the detection unit 6a is not executed (step). S58; No). When the control unit 8 determines that the selection change of the detection unit 6a is not executed (step S58; No), the process returns to the flow of FIG. 33 (step S49). Accordingly, it is possible to generate an image in which the detecting unit 6a to be selected is changed.

In the above-mentioned observation method, the microscope 721 according to the second modification may be used instead of the microscope 731 according to the third modification. According to the microscopes 721 and 731 according to the second and third modified examples and the observation method using the microscopes 721 and 731, it is possible to perform the process corresponding to the component separation of the third embodiment. Thus, by executing the phase shift process, the effective PSFs of the images obtained for each detection unit 6a can be aligned. Therefore, the image processing unit 7 adds the images of the respective detection units 6a obtained by executing the phase shift process to obtain an image with high resolution (eg, super-resolution image) while ensuring S/N. Can be generated. The fluorescent substance contained in the sample S is multiphoton excited by the excitation light L1. As a result, similar to the first embodiment, the image data on the sample surface Sa set in an arbitrary portion including the inside of the sample S can be acquired with high accuracy.

The technical scope of the present invention is not limited to the embodiments described in the above-described embodiments. One or more of the requirements described in the above embodiments and the like may be omitted. In addition, the requirements described in the above-described embodiments can be combined as appropriate. In addition, to the extent permitted by law, the disclosure of all documents cited in the above-mentioned embodiments and the like shall be incorporated as part of the description in the main text.

Further, in the above-described various embodiments and modified examples, the interference fringes formed by interference have been described as an example of striped illumination (stripe illumination), but the striped illumination (stripe illumination) formed by interference is not. Alternatively, striped illumination (stripe illumination) formed by a method other than interference may be used. Note that the “stripes” and “stripes” have a bright portion and a dark portion, and have a predetermined interval (predetermined period) between the bright portion and the bright portion or the dark portion and the dark portion. "Striped illumination (stripe illumination)" and "interference fringes" are formed on the sample surface.

DESCRIPTION OF SYMBOLS 1 microscope 3 light source 4 illumination optical system 5 detection optical system 6 detection device 6a multiple detection parts 7 image processing part 101 microscope (fifth embodiment) 106 detection device 201 microscope (sixth embodiment) 301 microscope (seventh embodiment) )
401 Microscope (Eighth Embodiment) 404 Illumination Optical System 501 Microscope (Ninth Embodiment) 601 Microscope (10th Embodiment)
701 Microscope (11th Embodiment)
711 Microscope (first modification) 721 Microscope (second modification)
731 microscope (third modification)
L2 interference fringe

Claims (28)

1. An illumination optical system having a scanning unit that scans a striped illumination, which is a striped illumination generated by light for exciting a fluorescent substance contained in a sample, in a plurality of directions of the sample.
   A detection optical system on which fluorescence from the sample generated by multiphoton excitation is incident,
   A detection device having a plurality of detection units for detecting fluorescence from the sample via the detection optical system,
   A microscope comprising: an image processing unit that generates an image using detection results of at least two detection units of the plurality of detection units.
2. The microscope according to claim 1, wherein the striped illumination has three or more bright portions in a cycle direction of the striped illumination.
3. The illumination optical system includes a luminous flux dividing portion that divides incident light into a plurality of luminous fluxes and an objective lens.
   The luminous flux dividing portion has an opening member having a plurality of openings.
   The microscope according to claim 1, wherein the aperture member is arranged in a pupil plane of the objective lens, in the vicinity of the pupil plane, in a pupil conjugate plane, or in the vicinity of the pupil conjugate plane.
4. The illumination optical system has a light beam splitting unit that splits incident light into a plurality of light beams,
   The luminous flux dividing portion has a diffraction grating and
   The microscope according to claim 1 or 2, wherein the diffraction grating is arranged at a position conjugate with the sample or near a position conjugate with the sample.
5. The microscope according to any one of claims 1 to 4, wherein the detection device has a line sensor in which the plurality of detection units are arranged in one direction.
6. The microscope according to any one of claims 1 to 4, wherein the detection device has an image sensor in which the plurality of detection units are arranged in two directions.
7. The microscope according to any one of claims 1 to 6, further comprising a striped direction changing portion for changing the direction of the striped illumination with respect to the sample.
8. The microscope according to any one of claims 1 to 7, further comprising an image rotation unit that rotates an image of the sample with respect to the plurality of detection units around an optical axis of the detection optical system.
9. The microscope according to claim 8, wherein the image rotating unit is arranged in an optical path that does not overlap with the illumination optical system in the detection optical system.
10. A fringe direction changing portion that changes the direction of the fringe illumination with respect to the sample,
    An image rotation unit that rotates the image of the sample with respect to the plurality of detection units around the optical axis of the detection optical system,
    The microscope according to any one of claims 1 to 6, wherein the striped direction changing portion and the image rotating portion are formed of the same member.
11. The microscope according to any one of claims 1 to 10, further comprising a polarization adjusting unit for adjusting the polarization state of the light when it is incident on the sample.
12. The microscope according to any one of claims 1 to 11, wherein the positions of the plurality of detection units are set based on the magnification of the detection optical system and the period of the fringe illumination.
13. The image processing unit generates an image using the detection results of the at least two detection units selected from the plurality of detection units based on the magnification of the detection optical system and the cycle of the fringe illumination. The microscope according to any one of claims 1 to 12.
14. The image processing unit according to any one of claims 1 to 13, which corrects data obtained from at least two detection units among the plurality of detection units based on the positions of the at least two detection units. microscope.
15. The microscope according to any one of claims 1 to 11, wherein the image processing unit converts data obtained from at least two detection units of the plurality of detection units into data in a frequency space.
16. The microscope according to claim 15, wherein the image processing unit converts data obtained from at least two detection units of the plurality of detection units into data on the frequency space by Fourier transform.
17. The microscope according to claim 15 or 16, wherein the image processing unit filters the data in the frequency space to generate the image.
18. 18. The image processing unit according to claim 15, wherein the image processing unit separates the data on the frequency space into a plurality of regions of the frequency space to generate a plurality of components to generate the image. Microscope.
19. The microscope according to claim 18, wherein the image processing unit separates the data in the frequency space into a plurality of regions in the frequency space based on a light intensity distribution of the interference fringes.
20. The microscope according to claim 18 or 19, wherein the plurality of areas are set so as not to overlap each other.
21. The illumination optical system includes a phase modulation element that changes the phase of the fringe illumination,
    The image processing unit generates the image by separating the data of the detection result detected by the detection unit in the frequency space into a plurality of components when the fringe illumination is in each of a plurality of phase states. The microscope according to any one of claims 15 to 17.
22. The microscope according to any one of claims 18 to 21, wherein the image processing unit converts the phase of at least part of data obtained by separating the plurality of components to generate the image.
23. The microscope according to claim 22, wherein the image processing unit determines the amount of phase conversion based on the positions of at least two detection units of the plurality of detection units and the light intensity distribution of the fringe illumination.
24. The microscope according to claim 22 or 23, wherein the image processing unit corrects the phase-converted data based on the position of the detection unit to generate the image.
25. 21. The image processing unit according to claim 1, wherein the image processing unit performs deconvolution on data obtained from at least two detection units of the plurality of detection units to generate the image. microscope.
26. The image processing unit performs the deconvolution on the data obtained from at least two detection units of the plurality of detection units based on the positions of the at least two detection units and the light intensity distribution of the fringe illumination. The microscope according to claim 25.
27. The microscope according to any one of claims 1 to 26, wherein the fringe illumination is an interference fringe.
28. Scanning in a plurality of directions of the sample fringe illumination is a striped illumination generated by light for multiphoton excitation of the fluorescent material contained in the sample,
    Detecting fluorescence from the sample generated by multiphoton excitation using a plurality of detectors,
    Generating an image using the detection results of at least two detectors of the plurality of detectors.

Images