CN111077121B - Rapid method and system for directly reconstructing structured light illumination super-resolution image in space domain - Google Patents

Rapid method and system for directly reconstructing structured light illumination super-resolution image in space domain Download PDF

Info

Publication number
CN111077121B
CN111077121B CN201911238624.6A CN201911238624A CN111077121B CN 111077121 B CN111077121 B CN 111077121B CN 201911238624 A CN201911238624 A CN 201911238624A CN 111077121 B CN111077121 B CN 111077121B
Authority
CN
China
Prior art keywords
light
super
structured light
resolution image
spatial
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201911238624.6A
Other languages
Chinese (zh)
Other versions
CN111077121A (en
Inventor
姚保利
但旦
雷铭
周兴
千佳
赵天宇
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
MGI Tech Co Ltd
Original Assignee
XiAn Institute of Optics and Precision Mechanics of CAS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by XiAn Institute of Optics and Precision Mechanics of CAS filed Critical XiAn Institute of Optics and Precision Mechanics of CAS
Priority to CN201911238624.6A priority Critical patent/CN111077121B/en
Publication of CN111077121A publication Critical patent/CN111077121A/en
Application granted granted Critical
Publication of CN111077121B publication Critical patent/CN111077121B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/62—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light
    • G01N21/63—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light optically excited
    • G01N21/64—Fluorescence; Phosphorescence
    • G01N21/645—Specially adapted constructive features of fluorimeters
    • G01N21/6456—Spatial resolved fluorescence measurements; Imaging
    • G01N21/6458—Fluorescence microscopy
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/0004—Microscopes specially adapted for specific applications
    • G02B21/002—Scanning microscopes
    • G02B21/0024—Confocal scanning microscopes (CSOMs) or confocal "macroscopes"; Accessories which are not restricted to use with CSOMs, e.g. sample holders
    • G02B21/0052—Optical details of the image generation
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/0004—Microscopes specially adapted for specific applications
    • G02B21/002—Scanning microscopes
    • G02B21/0024—Confocal scanning microscopes (CSOMs) or confocal "macroscopes"; Accessories which are not restricted to use with CSOMs, e.g. sample holders
    • G02B21/0052—Optical details of the image generation
    • G02B21/0076—Optical details of the image generation arrangements using fluorescence or luminescence

Abstract

The invention relates to a rapid method and a rapid system for directly reconstructing a structured light illumination super-resolution image in a space domain, which can be widely applied to the research in the fields of biology, medicine, material science, microelectronics and the like. Compared with the traditional frequency domain SIM super-resolution image reconstruction method, the method disclosed by the invention has the advantages that the operation is completely performed in a space domain, the processes of frequency spectrum separation, frequency spectrum movement, frequency spectrum fusion and the like are avoided, the time-consuming multiple forward/inverse Fourier transform operation is avoided, and the problem of artifact noise caused by frequency spectrum operation can be completely avoided; the demodulation matrix for super-resolution reconstruction can be generated in advance, the related operations are only multiplication and addition operations, and the process is simple and easy to implement, so that the reconstruction speed of the SIM super-resolution image is greatly improved.

Description

Rapid method and system for directly reconstructing structured light illumination super-resolution image in space domain
Technical Field
The invention relates to a rapid method for directly reconstructing a structured light illumination super-resolution image in a space domain, which can be widely applied to the research in the fields of biology, medicine, material science, microelectronics and the like.
Background
The spatial resolution of conventional optical microscopes is limited by the diffraction limit of light, which can only reach half the wavelength of light, and greatly limits the application range of optical microscopes, for example, in imaging structures in living biological cells. Achieving high spatial resolution imaging has been one of the important research topics in the field of optical microscopy. By labeling the sample with fluorescent molecules, the imaging signal-to-noise ratio and contrast of the optical microscopy technique are significantly improved. Based on the intensity response mechanism of fluorescent molecules to illumination light, various super-resolution fluorescence Microscopy techniques have been proposed, such as Photo-Activation Localization Microscopy (PALM), Stochastic Optical Reconstruction Microscopy (fluorescence Microscopy,short for STORM), Stimulated Emission Depletion fluorescence Microscopy (STED), Structured light Illumination fluorescence Microscopy (SIM), etc. The highest resolution of the current super-resolution fluorescence microscopic imaging technology is close to the resolution level of an electron microscope, provides a powerful tool for modern biomedicine, and also pushes related research to a new depth. In the super-resolution fluorescence microscopy technology, the SIM has the advantages of high imaging speed, no special requirement on fluorescent molecules, photobleaching, low phototoxicity and the like, and is particularly suitable for long-time dynamic observation of living cells under high resolution. SIMs can be classified into linear SIMs and non-linear SIMs based on the linear and non-linear responses of fluorescence to illumination light. The linear SIM adopts a structural light field with cosine distribution of intensity to illuminate the sample, the distribution of the excited fluorescence light field is in linear relation with the illumination light field, and the spatial resolution exceeds twice of the diffraction limit. Since the spatial resolution of the microscope system depends on the maximum spatial frequency f it can collect0And f is0Depending on the optical transfer function OTF of the system. When the sample contains high frequency information f>f0The details of the sample cannot be distinguished. If a spatial frequency f is used1The cosine fringe structure light illuminates the sample to generate a spatial frequency fm＝|f-f1Low frequency moire of l. Moire fringes are actually beat frequency signals of sample and structure light fields, and contain high frequency information f of sample super-diffraction resolution. Due to fm<f0The moire fringes can be recorded by a microscope system, and then the high-frequency information f of the sample can be extracted through algorithm demodulation, so that a high-resolution image of the sample is reconstructed. The nonlinear SIM is developed on the basis of the theory of the linear SIM, and by utilizing the nonlinear response characteristic of fluorescent molecules to the illumination light intensity, a fluorescence light field generated by excitation shows trapezoidal wave or square wave distribution with higher-order frequency, the formed moire fringe contains more high-frequency components of a sample, and the details of the finally reconstructed sample are richer. The super-resolution image reconstruction method is the core technology of the SIM. Current image reconstruction methods process data mainly in the spatial frequency domain (abbreviated FDR method), which is the method of image reconstructionThe defects mainly lie in that: firstly, Fourier transform, spectrum separation, spectrum movement, spectrum fusion, inverse Fourier transform and other processes are needed, and the steps are various; secondly, the reconstruction is very time-consuming due to multiple forward/inverse fourier transform operations. In order to reconstruct a super-resolution image, a linear SIM generally needs to acquire 6 or 9 structured light illumination images for operation, while a non-linear SIM needs more. As image size increases, reconstruction time consumption becomes more significant. Finally, operating (separating, moving, fusing) on the frequency components of the image data is easy to bring artifact noise, and the real details of the sample are covered. Due to the defects of the traditional super-resolution image reconstruction method, the current SIM system is difficult to realize real-time super-resolution dynamic observation of samples, and the application of the SIM in living subcellular dynamic observation is hindered.
Disclosure of Invention
Aiming at the problem that the super-resolution microscopy real-time dynamic observation is difficult due to complex operation and serious time consumption of the current structured light illumination super-resolution image reconstruction method, the invention provides a rapid method (SDR method for short) for directly reconstructing a structured light illumination super-resolution image in a space domain, the operation steps are greatly simplified compared with the traditional method, the steps are reduced from 5 traditional steps to 2 steps, and basic multiplication and addition operation is carried out without Fourier transform operation, so the reconstruction speed is greatly improved.
The technical scheme of the invention is to provide a rapid method for directly reconstructing a structured light illumination super-resolution image in a space domain, which comprises the following steps under the condition of linear excitation response:
step 1), generation and phase shift of a structured light field:
the light is illuminated by a laser or a Light Emitting Diode (LED) light source and modulated by a Spatial Light Modulator (SLM) or a Digital Micromirror Device (DMD) light field modulator, and a structured light illumination light field with light intensity meeting one-dimensional cosine function distribution is formed on a sample; the intensity of the structured light illumination field satisfies the distribution of formula (1):
Figure GDA0002640180350000031
where r denotes the two-dimensional object coordinate of the sample plane, I0Representing the mean value of the intensity of the structured light field, m representing the degree of modulation, k0The spatial frequency is represented by a representation of,
Figure GDA0002640180350000032
representing an initial phase;
controlling the light field modulation device to generate 3 structural light fields with different spatial directions and an included angle theta of 120 degrees between the adjacent directions in a sample plane; in each spatial direction by the amount of phase shift
Figure GDA0002640180350000033
Generating 3 structured light fields with different phase shifts, the phases of which are respectively
Figure GDA0002640180350000034
Generating 9 structured light fields at different positions in sequence for illuminating and exciting the sample to generate a fluorescence signal;
step 2), collecting a fluorescence image by an area array digital camera:
the method comprises the following steps that (1) an illumination light field with 3 phase shift structures corresponding to 3 spatial directions is imaged by a fluorescence microscope system, 9 fluorescence images illuminated by structural light are sequentially obtained by recording by an area array digital camera CMOS or CCD on an image surface of the imaging system, the fluorescence images are divided into 3 groups according to the spatial directions, each group comprises 3 different phases and are respectively recorded as: { D11(r’)，D12(r’)，D13(r’)}、{D21(r’)，D22(r’)，D23(r’)}、{D31(r’)，D32(r’)，D33(r ') }, where r' represents image plane two-dimensional image coordinates;
step 3), super-resolution image reconstruction processing:
step 3.1), according to the light intensity average value I of the structured light field0Degree of modulation m, spatial frequency k0Initial phase in 3 spatial directions
Figure GDA0002640180350000035
And a phase shift amount of 2 pi/3 to generate a demodulation system shown in formula (2)The number matrix is also correspondingly divided into 3 groups according to the spatial direction, each group contains 3 different phases which are respectively recorded as: { c11(r’)，c12(r’)，c13(r’)}、{c21(r’)，c22(r’)，c23(r’)}、{c31(r’)，c32(r’)，c33(r’)}；
Figure GDA0002640180350000041
Step 3.2), calculating the 9 structural light illuminated fluorescence images collected in the step 2) and the demodulation coefficient matrix generated in the step 3.1) according to the sequence by the formula (3) to obtain an initial super-resolution image RSDR(r’)：
Figure GDA0002640180350000042
Step 3.3), generating a new super-resolution equivalent point spread function P (r ') according to a formula (4) by using an optical system point spread function H (r') obtained theoretically or experimentally:
P(r’)＝H(r’)[1+cos(2πk0r’)] (4)
step 3.4), using the initial super-resolution image R obtained in step 3.2)SDR(R ') and the super-resolution equivalent point spread function P (R') generated in the step 3.3), and the deconvolution operation is completed to obtain the final super-resolution image RSIM(r’)。
The invention also provides a rapid system for directly reconstructing the structured light illumination super-resolution image in the airspace, which comprises a processor and a memory and is characterized in that: the memory stores a computer program which, when executed on the processor, performs the method of step 3) above.
The invention also provides a computer-readable storage medium, which is characterized in that: a computer program is stored which, when executed, implements the method described above in step 3).
The invention also provides another rapid method for directly reconstructing the structured light illumination super-resolution image in the space domain, which comprises the following steps under the condition of linear excitation response:
step 1), generation and phase shift of a structured light field:
the light is illuminated by a laser or a Light Emitting Diode (LED) light source, and a structured light illumination light field with light intensity meeting one-dimensional cosine function distribution is formed on a sample through the modulation of a Spatial Light Modulator (SLM) or a Digital Micromirror Device (DMD) light field modulation device; the intensity of the structured light illumination field satisfies the distribution of formula (1):
Figure GDA0002640180350000051
where r denotes the two-dimensional object coordinate of the sample plane, I0Representing the mean value of the intensity of the structured light field, m representing the degree of modulation, k0The spatial frequency is represented by a representation of,
Figure GDA0002640180350000052
representing an initial phase;
controlling the light field modulation device to generate 2 structured light fields with different spatial directions and an included angle theta of 90 degrees between the adjacent directions in a sample plane; in each spatial direction by the amount of phase shift
Figure GDA0002640180350000053
Generating 3 structured light fields with different phase shifts, the phases of which are respectively
Figure GDA0002640180350000054
Generating structural light fields at 6 different positions in sequence for illuminating and exciting the sample to generate a fluorescence signal;
step 2), collecting a fluorescence image by an area array digital camera:
the method comprises the following steps that (1) an illumination light field with 3 phase shift structures corresponding to 2 spatial directions is imaged by a fluorescence microscope system, 6 fluorescence images illuminated by structural light are sequentially recorded on an image surface of the imaging system by an area array digital camera CMOS or CCD, the fluorescence images are divided into 2 groups according to the spatial directions, each group comprises 3 different phases and are respectively recorded as: { D11(r’)，D12(r’)，D13(r’)}、{D21(r’)，D22(r’)，D23(r’)}；
Step 3), super-resolution image reconstruction processing:
step 3.1), according to the light intensity average value I of the structured light field0Degree of modulation m, spatial frequency k 02 initial phase in spatial direction
Figure GDA0002640180350000055
And phase shift amount pi/2, generating a demodulation coefficient matrix shown in formula (5), and correspondingly dividing the demodulation coefficient matrix into 2 groups according to the spatial direction, wherein each group contains 3 different phases which are respectively recorded as: { c11(r’)，c12(r’)，c13(r’)}、{c21(r’)，c22(r’)，c23(r’)}；
Figure GDA0002640180350000056
Step 3.2), calculating the 6 structural light illuminated fluorescence images collected in the step 2) and the demodulation coefficient matrix generated in the step 3.1) according to the sequence by a formula (6) to obtain an initial super-resolution image RSDR(r’)：
Figure GDA0002640180350000061
Step 3.3), generating a new super-resolution equivalent point spread function P (r ') according to a formula (4) by using an optical system point spread function H (r') obtained theoretically or experimentally;
P(r’)＝H(r’)[1+cos(2πk0r’)] (4)
step 3.4), using the initial super-resolution image R obtained in step 3.2)SDR(R ') and the super-resolution equivalent point spread function P (R') generated in the step 3.3), and the deconvolution operation is completed to obtain the final super-resolution image RSIM(r’)。
The invention also provides a rapid system for directly reconstructing the structured light illumination super-resolution image in the airspace, which comprises a processor and a memory and is characterized in that: the memory stores a computer program which, when executed on the processor, performs the method of step 3) above.
The invention also provides a computer-readable storage medium, which is characterized in that: a computer program is stored which, when executed, implements the method described above in step 3).
The invention also provides another rapid method for directly reconstructing the structured light illuminated super-resolution image in the space domain, which comprises the following steps under the condition of nonlinear excitation response:
step 1), excitation and phase shift of a nonlinear structure light field:
the light is illuminated by a laser or a Light Emitting Diode (LED) light source, and a structured light illumination light field with light intensity meeting one-dimensional cosine function distribution is formed on a sample through the modulation of a Spatial Light Modulator (SLM) or a Digital Micromirror Device (DMD) light field modulation device; the intensity of the structured light illumination field satisfies the distribution of formula (1);
Figure GDA0002640180350000062
where r denotes the two-dimensional object coordinate of the sample plane, I0Representing the mean value of the intensity of the structured light field, m representing the degree of modulation, k0The spatial frequency is represented by a representation of,
Figure GDA0002640180350000063
representing an initial phase;
by utilizing the nonlinear response characteristic of the fluorescent molecules, the intensity of the generated fluorescent signal light has the optical field distribution described by the formula (7):
Figure GDA0002640180350000071
where r denotes the two-dimensional object coordinate of the sample plane, I0Representing the mean value of the light intensity, M the harmonic order of the structured light field, bnIndicating the degree of modulation, k, of the nth harmonic0Which represents the spatial frequency of the fundamental frequency,
Figure GDA0002640180350000072
representing an initial phase;
controlling the optical field modulation device to generate (2M +1) structural optical fields with different spatial directions and an included angle theta between adjacent directions being 360 degrees/2M +1 on a sample plane; in each spatial direction, it is necessary to shift the phase by the amount
Figure GDA0002640180350000073
Generating (2M +1) structured light fields with different phase shifts; in the case of harmonic order M being 2, the light intensity distribution of the fluorescence signal generated by the nonlinear excitation response satisfies the structural light field described by formula (8):
Figure GDA0002640180350000074
in this case, it is necessary to generate 5 structured light fields with different spatial directions and an angle θ between adjacent directions of 72 ° on the sample plane; in each spatial direction, it is necessary to shift the phase by the amount
Figure GDA0002640180350000075
Generating 5 structured light fields with different phase shifts; so as to generate 25 excited fluorescence structure light fields at different positions;
2) acquiring a fluorescence image by an area array digital camera:
after imaging by a fluorescence microscope system, 25 fluorescence images illuminated by structural light are sequentially obtained on an image plane of the imaging system by an area array digital camera CMOS or CCD, the fluorescence images are divided into 5 groups according to the spatial direction, each group contains 5 different phases and are respectively marked as: { D11(r’)，D12(r’)，……，D15(r’)}、{D21(r’)，D22(r’)，……，D25(r’)}、……、{D51(r’)，D52(r’)，……，D55(r ') }, where r' represents image plane two-dimensional image coordinates;
step 3), super-resolution image reconstruction processing:
step 3.1), according to the light intensity average value I of the structured light field0Modulation b of each harmonic wave, M20、b1、b2Fundamental frequency k of space0Initial phase in 5 spatial directions
Figure GDA0002640180350000081
And phase shift amount 2 pi/5, obtaining a demodulation coefficient matrix described by formula (9), and correspondingly dividing the demodulation coefficient matrix into 5 groups according to the spatial direction, wherein each group contains 5 different phases which are respectively recorded as: { c11(r’)，c12(r’)，……，c15(r’)}、{c21(r’)，c22(r’)，……，c25(r’)}、……、{c51(r’)，c52(r’)，……，c55(r’)}：
Figure GDA0002640180350000082
Step 3.2), calculating the 25 structured light illuminated fluorescence images acquired in the step 2) and the demodulation coefficient matrix generated in the step 3.1) according to the sequence through a formula (10) to obtain an initial super-resolution image RSDR(r’)：
Figure GDA0002640180350000083
Step 3.3), generating a new super-resolution equivalent point spread function P (r ') according to a formula (11) by using an optical system point spread function H (r') obtained theoretically or experimentally:
P(r’)＝H(r’)×{b0+b1cos[2πk0r’]+b2cos[2(2πk0r’)]} (11)
step 3.4), using the initial super-resolution image R obtained in step 3.2)SDR(R ') and the super-resolution equivalent point spread function P (R') generated in the step 3.3), and the deconvolution operation is completed to obtain the final super-resolution image RSIM(r’)。
The invention also provides a rapid system for directly reconstructing the structured light illumination super-resolution image in the airspace, which comprises a processor and a memory and is characterized in that: the memory stores a computer program which, when executed on the processor, performs the method of step 3) above.
The invention also provides a computer-readable storage medium, which is characterized in that: a computer program is stored which, when executed, implements the method described above in step 3).
The invention also provides another rapid method for directly reconstructing the structured light illuminated super-resolution image in the space domain, which comprises the following steps under the condition of nonlinear excitation response:
step 1), excitation and phase shift of a nonlinear structure light field:
the light is illuminated by a laser or a Light Emitting Diode (LED) light source, and a structured light illumination light field with light intensity meeting one-dimensional cosine function distribution is formed on a sample through the modulation of a Spatial Light Modulator (SLM) or a Digital Micromirror Device (DMD) light field modulation device; the intensity of the structured light illumination field satisfies the distribution of formula (1);
Figure GDA0002640180350000091
where r denotes the two-dimensional object coordinate of the sample plane, I0Representing the mean value of the intensity of the structured light field, m representing the degree of modulation, k0The spatial frequency is represented by a representation of,
Figure GDA0002640180350000092
representing an initial phase;
by utilizing the nonlinear response characteristic of the fluorescent molecules, the intensity of the generated fluorescent signal light has the optical field distribution described by the formula (7):
Figure GDA0002640180350000093
where r denotes the two-dimensional object coordinate of the sample plane, I0Representing mean value of light intensity, M representing structured light fieldHarmonic order, bnIndicating the degree of modulation, k, of the nth harmonic0Which represents the spatial frequency of the fundamental frequency,
Figure GDA0002640180350000094
representing an initial phase;
in the case of harmonic order M being 3, the light intensity distribution of the fluorescence signal generated by the nonlinear excitation response satisfies the structural light field described by formula (12):
Figure GDA0002640180350000095
Figure GDA0002640180350000101
controlling the light field modulation device to generate 7 structured light fields with different spatial directions and an included angle theta of 51.4 degrees between the adjacent directions on a sample plane; in each spatial direction, it is necessary to shift the phase by the amount
Figure GDA0002640180350000102
Generating 7 structured light fields with different phase shifts; so as to generate 49 light fields of the excited fluorescence structure at different positions;
step 2), collecting a fluorescence microscopic image by an area array digital camera:
after 7 phase shift structure light fields corresponding to 7 spatial directions are imaged by a fluorescence microscope system, 49 fluorescence images illuminated by structural light are sequentially recorded and obtained on an image surface of the imaging system by an area array digital camera CMOS or CCD, the fluorescence images are divided into 7 groups according to the spatial directions, and each group comprises 7 different phases which are respectively recorded as: { D11(r’)，D12(r’),……,D17(r’)}、{D21(r’)，D22(r’),……,D27(r’)}、……、{D71(r’)，D72(r’),……,D77(r ') }, where r' represents image plane two-dimensional image coordinates;
step 3), super-resolution image reconstruction processing:
step 3.1), rootAccording to the mean value of the light intensity of the structured light field I0Modulation b of each harmonic wave 30、b1、b2、b3Fundamental frequency k of space0Initial phase in 7 spatial directions
Figure GDA0002640180350000103
Figure GDA0002640180350000104
And phase shift amount 2 pi/7, obtaining a demodulation coefficient matrix described by formula (13), and correspondingly dividing the demodulation coefficient matrix into 7 groups according to the space direction, wherein each group contains 7 different phases which are respectively recorded as: { c11(r’)，c12(r’),……,c17(r’)}、{c21(r’)，c22(r’),……,c27(r’)}、……、{c71(r’)，c72(r’),……,c77(r’)}：
Figure GDA0002640180350000111
Step 3.2), calculating the 49 structural light illuminated fluorescence images acquired in the step 2) and the demodulation coefficient matrix generated in the step 3.1) according to the sequence by a formula (14) to obtain an initial super-resolution image RSDR(r’)：
Figure GDA0002640180350000112
Step 3.3), generating a new super-resolution equivalent point spread function P (r ') according to a formula (15) by using an optical system point spread function H (r') obtained theoretically or experimentally:
P(r’)＝H(r’)×{b0+b1cos[2πk0r’]+b2cos[2(2πk0r’)]+b3cos[3(2πk0r’)]} (15)
step 3.4), using the initial super-resolution image R obtained in step 3.2)SDR(r ') and the super-resolution equivalent point spread function P (r') generated in the step 3.3), and the deconvolution operation is completed to obtainFinal super-resolution image RSIM(r’)。
The invention also provides a rapid system for directly reconstructing the structured light illumination super-resolution image in the airspace, which comprises a processor and a memory and is characterized in that: the memory stores a computer program which, when executed on the processor, performs the method of step 3) above.
The invention also provides a computer-readable storage medium, which is characterized in that: a computer program is stored which, when executed, implements the method described above in step 3).
The invention has the advantages that:
the invention directly and rapidly reconstructs the SIM super-resolution image in a space domain. Compared with the traditional frequency domain SIM super-resolution image reconstruction method, the method disclosed by the invention has the advantages that the operation is completely carried out in a space domain, the processes of frequency spectrum separation, frequency spectrum movement, frequency spectrum fusion and the like are avoided, the time-consuming multiple forward/inverse Fourier transform operations are avoided, and the problem of artifact noise caused by frequency spectrum operation can be completely avoided; the demodulation matrix for super-resolution reconstruction can be generated in advance, the related operations are only multiplication and addition operations, and the process is simple and easy to implement, so that the reconstruction speed of the SIM super-resolution image is greatly improved.
Drawings
FIG. 1 illustrates the operation steps and comparison of the present invention method and conventional method for super-resolution image reconstruction;
FIGS. 2a and 2b are graphs showing the calculation time consumption statistics and comparison of super-resolution image reconstruction by the method of the present invention and the conventional method in different image sizes;
FIG. 3 is a light path diagram of a projection type SIM super-resolution microscope system based on digital micromirror DMD modulation and LED illumination;
the reference numbers in the figures are: 1-LED illumination light source, 2-beam splitting prism, 3-structured light generator, 4-collimating lens, 5-illumination light filter, 6-beam splitter, 7-reflector, 8-microscope objective, 9-objective stage, 10-filter, 11-tube lens and 12-area array digital camera;
FIG. 4 is a light path diagram of an interferometric SIM super-resolution microscope system based on spatial light modulator SLM modulation and zero-order vortex half-wave plate polarization control;
the reference numbers in the figures are: the system comprises a 1-laser illumination source, 2, 3, 7, 10, 11-lenses, a 4-polarization beam splitter, a 5-half wave plate, a 6-spatial light modulator SLM, an 8-spatial filter, a 9-zero-order vortex half wave plate, a 12-beam splitter, a 13-microscope objective, a 14-objective table, a 15-optical filter, a 16-tube lens and a 17-area array digital camera.
Fig. 5 is a three-channel super-resolution microscopic image of bovine pulmonary artery endothelial cells (BPAE) obtained using a DMD modulation and LED illumination based SIM super-resolution microscopic system. The blue channel is a cell nucleus image under the excitation of 405nm light, the green channel is an actin image under the excitation of 470nm light, and the red channel is a mitochondria image under the excitation of 565nm light; wherein (a) is a normal wide-field fluorescence image of the sample, (b) is a structured light illuminated three-channel super-resolution microscopy image reconstructed using a conventional method, (c) is a structured light illuminated three-channel super-resolution microscopy image reconstructed using the method of the present invention, (d) - (f) are enlarged views of the dashed box regions in (a) - (c);
FIG. 6 is a super-resolution microscopic image of mitochondria in bovine pulmonary artery endothelial cells obtained by using a Spatial Light Modulator (SLM) -based modulation and a zero-order vortex half-wave plate polarization control SIM super-resolution microscopic system. Wherein (a) is a normal wide-field fluorescence image, (b) is a structured light illuminated super-resolution microscopy image reconstructed using a conventional method, (c) is a structured light illuminated super-resolution microscopy image reconstructed using the method of the present invention, (d) - (f) are enlarged views of the dashed box regions in (a) - (c);
fig. 7 is a graph of the intensity distribution along the line marked in fig. 6(d) -6 (f).
Detailed Description
The invention is further described with reference to the following figures and specific embodiments.
The method can be implemented on a mainstream SIM super-resolution microscope system, and comprises a projection type SIM super-resolution microscope system based on digital micromirror DMD modulation and LED illumination, an interference type SIM super-resolution microscope system based on spatial light modulator SLM modulation and zero-order vortex half-wave plate polarization control, and the like. Compared with the traditional method, the operation steps are greatly simplified, as shown in the comparison of fig. 1, the traditional steps comprise 5 steps of Fourier transform, frequency spectrum separation, frequency spectrum movement, frequency spectrum fusion and inverse Fourier transform, while the invention reduces the traditional 5 steps into 2 steps, and the basic multiplication and addition operations are carried out, so that the Fourier transform operation is not needed, and the reconstruction speed is greatly improved.
A projection type SIM super-resolution microscope system based on digital micromirror DMD modulation and LED illumination is shown in fig. 3: the device comprises an LED illumination light source 1, a total internal reflection beam splitter prism 2 arranged on an illumination light path, a structured light generator 3 arranged on a reflection light path of the beam splitter prism 2, a collimating lens 4 and an illumination light filter 5 which are sequentially arranged on a transmission light path of the beam splitter prism 2, a beam splitter 6 arranged behind the illumination light filter 5, a reflector 7 arranged on a transmission light path of the beam splitter 6, a microscope objective 8 and an objective table 9 which are arranged on a light path behind the reflector 7, a light filter 10 and a barrel lens 11 which are sequentially arranged on a fluorescence reflection light path of the beam splitter 6, and an area array digital camera 12 arranged behind the barrel lens 11. The LED illumination light source 1 is an incoherent four-color LED light source, and the structured light generator 3 is a digital micromirror DMD.
An interferometric SIM super-resolution microscope system based on spatial light modulator SLM modulation and zero-order vortex half-wave plate polarization control is shown in figure 4: comprises a laser illumination light source 1, a beam expanding and collimating lens group (a lens 2 and a lens 3) arranged at the rear end of the laser illumination light source 1, a polarized beam splitter 4 arranged behind the beam expanding and collimating lens group, a half-wave plate 5 and a spatial light modulator SLM6 which are sequentially arranged on a transmission light path of the polarized beam splitter 4, a lens 7 arranged on a reflection light path of the polarized beam splitter 4, and a spatial filter 8 arranged at the rear end of the lens 7, the device comprises a zero-order vortex half-wave plate 9 arranged behind a spatial filter 8, a 4f system consisting of a lens 10 and a lens 11 arranged behind the zero-order vortex half-wave plate 9, a spectroscope 12 arranged behind the 4f system, a microscope objective 13 and an objective table 14 arranged on a transmission light path of the spectroscope 12, an optical filter 15 and a barrel mirror 16 arranged on a fluorescence reflection light path of the spectroscope 12, and an area array digital camera 17 arranged behind the barrel mirror 16. The spatial light modulator SLM6 is a reflective ferroelectric liquid crystal spatial light modulator.
Example one
The present embodiment is based on the super-resolution image reconstruction of the projection SIM with digital micromirror DMD modulation and LED illumination, and is implemented by the following steps:
step 1, using a projection type SIM super-resolution microscope system based on digital micromirror DMD modulation and LED illumination shown in figure 3, firstly, enabling an LED light beam with a wavelength of 405nm to enter a beam splitter prism 2 and irradiate the digital micromirror DMD, enabling structured light generated after the modulation of the digital micromirror DMD to penetrate through the beam splitter prism 2 to be emitted, enabling the structured light to enter a microscope objective 8 after being collimated by a collimating lens 4, and enabling the microscope objective 8 to micro-compress and project structured light stripes on a focal plane of the microscope objective 8;
step 2, placing the bovine pulmonary artery endothelial cell sample on an object stage 9 and adjusting the sample to a focal plane of a microscope objective, illuminating the sample by structured light generated by micro projection, and exciting fluorescent molecules of a labeled organelle to emit light;
step 3, controlling the digital micromirror DMD to load 2 light fields of 6 structures with the spatial direction included angle of 90 degrees and the phase in each spatial direction of 0, pi/2 and pi in sequence, and correspondingly acquiring 6 images by the area array digital camera 12 respectively, and recording as (D)11，D12，D13) And (D)21，D22，D23) And stored in the computer; firstly, according to the light intensity average value I of the structured light field0Degree of modulation m, spatial frequency k 02 initial phase in spatial direction
Figure GDA0002640180350000153
And phase shift amount pi/2, generating a demodulation coefficient matrix shown in formula (5), and correspondingly dividing the demodulation coefficient matrix into 2 groups according to the spatial direction, wherein each group contains 3 different phases which are respectively recorded as: { c11(r’)，c12(r’)，c13(r’)}、{c21(r’)，c22(r’)，c23(r’)}；
Figure GDA0002640180350000151
Secondly, the acquired 6 images and the generated demodulation coefficient matrix are calculated by a formula (6) in sequence to obtain an initial super-resolution image RSDR(r’)：
Figure GDA0002640180350000152
Then, generating a new super-resolution equivalent point spread function P (r ') according to a formula (4) by using an optical system point spread function H (r') obtained theoretically or experimentally;
P(r’)＝H(r’)[1+cos(2πk0r’)] (4)
finally, the obtained initial super-resolution image R is utilizedSDR(R ') and the generated super-resolution equivalent point spread function P (R') to complete the deconvolution operation and obtain the final super-resolution image RSIM(r’)。
Step 4, switching the wavelength of the LED to 470nm and 565nm in sequence, repeating the steps 1 to 3 under each wavelength, exciting fluorescence molecules for marking cell actin and mitochondria to emit light respectively, and obtaining super-resolution microscopic images of the cell, actin and mitochondria in sequence;
and 5, carrying out channel fusion on the obtained cell nucleus, actin and mitochondria super-resolution microscopic images to obtain a three-channel super-resolution microscopic image of the bovine pulmonary artery endothelial cell.
Fig. 5 is a three-channel super-resolution microscopic image of bovine pulmonary artery endothelial cells obtained using a projection SIM super-resolution microscopy system based on digital micromirror DMD modulation and LED illumination. The experiment used a 100X microscope objective with a numerical aperture NA of 1.49. (a) The method comprises the following steps of (a) obtaining a common wide-field fluorescence microscopic image of a bovine pulmonary artery endothelial cell sample, (b) obtaining a structured light illuminated three-channel super-resolution microscopic image reconstructed by using a traditional method, and (c) obtaining the structured light illuminated three-channel super-resolution microscopic image reconstructed by using the method. (d) - (f) is an enlarged view of the dashed-line frame region in (a) - (c). It can be seen that the quality of the images reconstructed by both methods is substantially the same. The reconstruction time of the super-resolution image of the method is 0.0447 seconds, while the reconstruction time of the traditional method is 3.747 seconds, and the reconstruction speed of the super-resolution image of the method is 84 times that of the traditional method.
The embodiment also provides a rapid system for directly reconstructing the structured light illumination super-resolution image in the space domain, which comprises a processor and a memory, wherein the memory stores a computer program, and the computer program executes the super-resolution image reconstruction processing method when running in the processor.
The present embodiment also provides a computer-readable storage medium storing a program that, when executed, implements the above-described super-resolution image reconstruction processing method. In some possible embodiments, the invention may also be implemented in the form of a program product comprising program code means for causing a terminal device to carry out the steps according to various exemplary embodiments of the invention described in the method part of the description above, when said program product is run on the terminal device.
A program product for implementing the above method, which may employ a portable compact disc read only memory (CD-ROM) and include program code, may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in the present invention, the computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
Example two
The embodiment is based on spatial light modulator SLM modulation and zero-order vortex half-wave plate polarization control, and the super-resolution image reconstruction of the interferometric SIM:
step 1, using an interferometric SIM super-resolution microscope system based on spatial light modulator SLM modulation and zero-order vortex half-wave plate polarization control shown in FIG. 4, laser with a wavelength of 532nm is incident on a polarization beam splitter 4 and vertically irradiates a spatial light modulator SLM6, and a vertically-polarized multi-stage diffraction beam is generated. The converged multi-order diffracted zero-order and high-order beams are blocked using the spatial filter 8, leaving only ± 1-order diffracted beams. Then, the plus or minus 1 st order diffraction light with the polarization state changed by the zero-order vortex half-wave plate 9 enters a microscope objective 13, and the two beams of diffraction light interfere with each other and form a structured light field on the focal plane of the objective for illuminating a sample;
step 2, placing the bovine pulmonary artery endothelial cell sample on an objective table, adjusting the bovine pulmonary artery endothelial cell sample to an objective focal plane, and exciting fluorescent molecules of marked cell mitochondria to emit light by using structured light illumination;
and 3, controlling the SLM to load 3 structured light fields with the spatial direction included angles of 120 degrees and the phases of 0, 2 pi/3 and 4 pi/3 in each spatial direction in sequence, and correspondingly acquiring 9 images by the area-array digital camera respectively, wherein the image is marked as (D)11，D12，D13)、(D21，D22，D23) And (D)31，D32，D33) And stored in the computer.
Step 4, firstly, according to the light intensity average value I of the structured light field0Degree of modulation m, spatial frequency k0Initial phase in 3 spatial directions
Figure GDA0002640180350000181
And the phase shift amount is 2 pi/3, a demodulation coefficient matrix shown in the formula (2) is generated, the demodulation coefficient matrix is also correspondingly divided into 3 groups according to the space direction, each group contains 3 different phases which are respectively recorded as: { c11(r’)，c12(r’)，c13(r’)}、{c21(r’)，c22(r’)，c23(r’)}、{c31(r’)，c32(r’)，c33(r’)}；
Figure GDA0002640180350000182
Secondly, illuminating the 9 pieces of structured light collected in the step 3) with fluorescenceThe image and the generated demodulation coefficient matrix are calculated by a formula (3) in sequence to obtain an initial super-resolution image RSDR(r’)：
Figure GDA0002640180350000183
Then, a new super-resolution equivalent point spread function P (r ') is generated according to formula (4) from the optical system point spread function H (r') obtained theoretically or experimentally:
P(r’)＝H(r’)[1+cos(2πk0r’)] (4)
finally, the obtained initial super-resolution image R is utilizedSDRAnd (r ') and the generated super-resolution equivalent point spread function P (r') to complete deconvolution operation and obtain a mitochondrial super-resolution image of the bovine pulmonary artery endothelial cell.
FIG. 6 is a super-resolution image of mitochondria within bovine pulmonary artery endothelial cells obtained using an interferometric SIM super-resolution microscopy system based on Spatial Light Modulator (SLM) modulation and zero-order vortex half-waveplate polarization control. The experiment used a 100X microscope objective with a numerical aperture NA of 1.49. (a) Is a common wide-field fluorescence image, (b) is a structured light illumination super-resolution image reconstructed by using a traditional method, (c) is a structured light illumination super-resolution image reconstructed by using the method of the invention, (d) - (f) are enlarged images of dotted line frame areas in (a) - (c); fig. 7 is a graph of the intensity distribution along the line marked in fig. 6(d) -6 (f). It can be seen that the quality of the images reconstructed by both methods is substantially the same. The reconstruction time of the super-resolution image of the method is 0.0934 seconds, while the reconstruction time of the traditional method is 4.7842 seconds, and the reconstruction speed of the super-resolution image of the method is 51 times that of the traditional method.
The embodiment also provides a rapid system for directly reconstructing the structured light illumination super-resolution image in the space domain, which comprises a processor and a memory, wherein the memory stores a computer program, and the computer program executes the super-resolution image reconstruction processing method when running in the processor.
The present embodiment also provides a computer-readable storage medium storing a program that, when executed, implements the above-described super-resolution image reconstruction processing method. In some possible embodiments, the invention may also be implemented in the form of a program product comprising program code means for causing a terminal device to carry out the steps according to various exemplary embodiments of the invention described in the method part of the description above, when said program product is run on the terminal device.
A program product for implementing the above method, which may employ a portable compact disc read only memory (CD-ROM) and include program code, may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in the present invention, the computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
In order to show the superiority of the method in the reconstruction time of the SIM super-resolution image compared with the traditional method, under the condition that the sizes of 4 common images are 512 multiplied by 512, 1024 multiplied by 1024, 1600 multiplied by 1600 and 2048 multiplied by 2048 pixels, the reconstruction time of the super-resolution image of 200 groups of SIM original images successively uses the method and the traditional method is counted. Each set of SIM original images consists of 9 structured light illumination images. As shown in FIGS. 2a and 2b, the mean value of the reconstruction time of the method is dozens of times less than that of the conventional method in any image size, and the statistical variance is one order of magnitude less, thus showing the rapidity and stability of the method. With the increase of the image size, the ratio of the reconstruction time average value of the method of the invention to the conventional method is gradually increased, and reaches 51.4 under 2048 × 2048 pixels, which shows that the reconstruction speed of the method of the invention is more than 50 times that of the conventional method. The software for calculation is Matlab R2013a, and the computer is configured by an Intel i7-4790K CPU, a 32GB memory and a Windows 7x64(SP1) operating system.

Claims (12)

1. A rapid method for directly reconstructing a structured light illumination super-resolution image in a space domain is characterized by comprising the following steps of:
step 1), generation and phase shift of a structured light field:
the light is illuminated by a laser or a Light Emitting Diode (LED) light source and modulated by a Spatial Light Modulator (SLM) or a Digital Micromirror Device (DMD) light field modulator, and a structured light illumination light field with light intensity meeting one-dimensional cosine function distribution is formed on a sample; the intensity of the structured light illumination field satisfies the distribution of formula (1):
Figure FDA0002640180340000011
where r denotes the two-dimensional object coordinate of the sample plane, I0Representing the mean value of the intensity of the structured light field, m representing the degree of modulation, k0The spatial frequency is represented by a representation of,
Figure FDA0002640180340000012
representing an initial phase;
controlling the light field modulation device to generate 3 structural light fields with different spatial directions and an included angle theta of 120 degrees between the adjacent directions in a sample plane; in each spatial direction by the amount of phase shift
Figure FDA0002640180340000013
Generating 3 structured light fields with different phase shifts, the phases of which are respectively
Figure FDA0002640180340000014
And
Figure FDA0002640180340000015
generating 9 structured light fields at different positions in sequence for illuminating and exciting the sample to generate a fluorescence signal;
step 2), collecting a fluorescence image by an area array digital camera:
the method comprises the following steps that (1) an illumination light field with 3 phase shift structures corresponding to 3 spatial directions is imaged by a fluorescence microscope system, 9 fluorescence images illuminated by structural light are sequentially obtained by recording by an area array digital camera CMOS or CCD on an image surface of the imaging system, the fluorescence images are divided into 3 groups according to the spatial directions, each group comprises 3 different phases and are respectively recorded as: { D11(r’)，D12(r’)，D13(r’)}、{D21(r’)，D22(r’)，D23(r’)}、{D31(r’)，D32(r’)，D33(r ') }, where r' represents image plane two-dimensional image coordinates;
step 3), super-resolution image reconstruction processing:
step 3.1), according to the light intensity average value I of the structured light field0Degree of modulation m, spatial frequency k0Initial phase in 3 spatial directions
Figure FDA0002640180340000021
And the phase shift amount is 2 pi/3, a demodulation coefficient matrix shown in the formula (2) is generated, the demodulation coefficient matrix is also correspondingly divided into 3 groups according to the space direction, each group contains 3 different phases which are respectively recorded as: { c11(r’)，c12(r’)，c13(r’)}、{c21(r’)，c22(r’)，c23(r’)}、{c31(r’)，c32(r’)，c33(r’)}；
Figure FDA0002640180340000022
Step 3.2), the 9 fluorescence images illuminated by the structured light collected in the step 2) and the step 3.1)The generated demodulation coefficient matrix is calculated by a formula (3) in sequence to obtain an initial super-resolution image RSDR(r’)：
Figure FDA0002640180340000023
Step 3.3), generating a new super-resolution equivalent point spread function P (r ') according to a formula (4) by using an optical system point spread function H (r') obtained theoretically or experimentally:
P(r’)=H(r’)[1+cos(2πk0r’)] (4)
step 3.4), using the initial super-resolution image R obtained in step 3.2)SDR(R ') and the super-resolution equivalent point spread function P (R') generated in the step 3.3), and the deconvolution operation is completed to obtain the final super-resolution image RSIM(r’)。
2. A fast system for directly reconstructing a structured light illumination super-resolution image in a space domain comprises a processor and a memory, and is characterized in that: the memory has stored therein a computer program which, when run on the processor, performs the method of step 3) of claim 1.
3. A computer-readable storage medium characterized by: a computer program is stored which, when executed, implements the method of claim 1, step 3).
4. A rapid method for directly reconstructing a structured light illumination super-resolution image in a space domain is characterized by comprising the following steps of:
step 1), generation and phase shift of a structured light field:
the light is illuminated by a laser or a Light Emitting Diode (LED) light source, and a structured light illumination light field with light intensity meeting one-dimensional cosine function distribution is formed on a sample through the modulation of a Spatial Light Modulator (SLM) or a Digital Micromirror Device (DMD) light field modulation device; the intensity of the structured light illumination field satisfies the distribution of formula (1):
Figure FDA0002640180340000031
where r denotes the two-dimensional object coordinate of the sample plane, I0Representing the mean value of the intensity of the structured light field, m representing the degree of modulation, k0The spatial frequency is represented by a representation of,
Figure FDA0002640180340000032
representing an initial phase;
controlling the light field modulation device to generate 2 structured light fields with different spatial directions and an included angle theta of 90 degrees between the adjacent directions in a sample plane; in each spatial direction by the amount of phase shift
Figure FDA0002640180340000033
Generating 3 structured light fields with different phase shifts, the phases of which are respectively
Figure FDA0002640180340000034
And
Figure FDA0002640180340000035
generating structural light fields at 6 different positions in sequence for illuminating and exciting the sample to generate a fluorescence signal;
step 2), collecting a fluorescence image by an area array digital camera:
the method comprises the following steps that (1) an illumination light field with 3 phase shift structures corresponding to 2 spatial directions is imaged by a fluorescence microscope system, 6 fluorescence images illuminated by structural light are sequentially recorded on an image surface of the imaging system by an area array digital camera CMOS or CCD, the fluorescence images are divided into 2 groups according to the spatial directions, each group comprises 3 different phases and are respectively recorded as: { D11(r’)，D12(r’)，D13(r’)}、{D21(r’)，D22(r’)，D23(r’)}；
Step 3), super-resolution image reconstruction processing:
step 3.1), according to the light intensity average value I of the structured light field0Modulation ofDegree m, spatial frequency k02 initial phase in spatial direction
Figure FDA0002640180340000036
And phase shift amount pi/2, generating a demodulation coefficient matrix shown in formula (5), and correspondingly dividing the demodulation coefficient matrix into 2 groups according to the spatial direction, wherein each group contains 3 different phases which are respectively recorded as: { c11(r’)，c12(r’)，c13(r’)}、{c21(r’)，c22(r’)，c23(r’)}；
Figure FDA0002640180340000041
Step 3.2), calculating the 6 structural light illuminated fluorescence images collected in the step 2) and the demodulation coefficient matrix generated in the step 3.1) according to the sequence by a formula (6) to obtain an initial super-resolution image RSDR(r’)：
Figure FDA0002640180340000042
Step 3.3), generating a new super-resolution equivalent point spread function P (r ') according to a formula (4) by using an optical system point spread function H (r') obtained theoretically or experimentally;
P(r’)=H(r’)[1+cos(2πk0r’)] (4)
step 3.4), using the initial super-resolution image R obtained in step 3.2)SDR(R ') and the super-resolution equivalent point spread function P (R') generated in the step 3.3), and the deconvolution operation is completed to obtain the final super-resolution image RSIM(r’)。
5. A fast system for directly reconstructing a structured light illumination super-resolution image in a space domain comprises a processor and a memory, and is characterized in that: the memory has stored therein a computer program which, when run on the processor, performs the method of step 3) of claim 4.
6. A computer-readable storage medium characterized by: a computer program is stored which, when executed, implements the method of step 3) of claim 4.
7. A rapid method for directly reconstructing a structured light illumination super-resolution image in a space domain is characterized by comprising the following steps under the condition of nonlinear excitation response:
step 1), excitation and phase shift of a nonlinear structure light field:
the light is illuminated by a laser or a Light Emitting Diode (LED) light source, and a structured light illumination light field with light intensity meeting one-dimensional cosine function distribution is formed on a sample through the modulation of a Spatial Light Modulator (SLM) or a Digital Micromirror Device (DMD) light field modulation device; the intensity of the structured light illumination field satisfies the distribution of formula (1);
Figure FDA0002640180340000051
where r denotes the two-dimensional object coordinate of the sample plane, I0Representing the mean value of the intensity of the structured light field, m representing the degree of modulation, k0The spatial frequency is represented by a representation of,
Figure FDA0002640180340000052
representing an initial phase;
by utilizing the nonlinear response characteristic of the fluorescent molecules, the intensity of the generated fluorescent signal light has the optical field distribution described by the formula (7):
Figure FDA0002640180340000053
where r denotes the two-dimensional object coordinate of the sample plane, I0Representing the mean value of the light intensity, M the harmonic order of the structured light field, bnIndicating the degree of modulation, k, of the nth harmonic0Which represents the spatial frequency of the fundamental frequency,
Figure FDA0002640180340000054
representing an initial phase;
controlling the optical field modulation device to generate (2M +1) structural optical fields with different spatial directions and an included angle theta between adjacent directions being 360 degrees/2M +1 on a sample plane; in each spatial direction, it is necessary to shift the phase by the amount
Figure FDA0002640180340000055
Generating (2M +1) structured light fields with different phase shifts; in the case of harmonic order M being 2, the light intensity distribution of the fluorescence signal generated by the nonlinear excitation response satisfies the structural light field described by formula (8):
Figure FDA0002640180340000056
in this case, it is necessary to generate 5 structured light fields with different spatial directions and an angle θ between adjacent directions of 72 ° on the sample plane; in each spatial direction, it is necessary to shift the phase by the amount
Figure FDA0002640180340000057
Generating 5 structured light fields with different phase shifts; so as to generate 25 excited fluorescence structure light fields at different positions;
2) acquiring a fluorescence image by an area array digital camera:
after imaging by a fluorescence microscope system, 25 fluorescence images illuminated by structural light are sequentially obtained on an image plane of the imaging system by an area array digital camera CMOS or CCD, the fluorescence images are divided into 5 groups according to the spatial direction, each group contains 5 different phases and are respectively marked as: { D11(r’)，D12(r’)，……，D15(r’)}、{D21(r’)，D22(r’)，……，D25(r’)}、……、{D51(r’)，D52(r’)，……，D55(r ') }, where r' represents image plane two-dimensional image coordinates;
step 3), super-resolution image reconstruction processing:
step 3.1), according to the light intensity average value I of the structured light field0Modulation b of each harmonic wave, M20、b1、b2Fundamental frequency k of space0Initial phase in 5 spatial directions
Figure FDA0002640180340000061
And phase shift amount 2 pi/5, obtaining a demodulation coefficient matrix described by formula (9), and correspondingly dividing the demodulation coefficient matrix into 5 groups according to the spatial direction, wherein each group contains 5 different phases which are respectively recorded as: { c11(r’)，c12(r’)，……，c15(r’)}、{c21(r’)，c22(r’)，……，c25(r’)}、……、{c51(r’)，c52(r’)，……，c55(r’)}：
Figure FDA0002640180340000062
Step 3.2), calculating the 25 structured light illuminated fluorescence images acquired in the step 2) and the demodulation coefficient matrix generated in the step 3.1) according to the sequence through a formula (10) to obtain an initial super-resolution image RSDR(r’)：
Figure FDA0002640180340000063
Step 3.3), generating a new super-resolution equivalent point spread function P (r ') according to a formula (11) by using an optical system point spread function H (r') obtained theoretically or experimentally:
P(r’)=H(r’)×{b0+bbcos[2πk0r’]+b2cos[2(2πk0r’)]} (11)
step 3.4), using the initial super-resolution image R obtained in step 3.2)SDR(R ') and the super-resolution equivalent point spread function P (R') generated in the step 3.3), and the deconvolution operation is completed to obtain the final super-resolution image RSIM(r’)。
8. A fast system for directly reconstructing a structured light illumination super-resolution image in a space domain comprises a processor and a memory, and is characterized in that: the memory has stored therein a computer program which, when run on the processor, performs the method of step 3) of claim 7.
9. A computer-readable storage medium characterized by: a computer program is stored which, when executed, implements the method of step 3) of claim 7.
10. A rapid method for directly reconstructing a structured light illumination super-resolution image in a space domain is characterized by comprising the following steps under the condition of nonlinear excitation response:
step 1), excitation and phase shift of a nonlinear structure light field:
the light is illuminated by a laser or a Light Emitting Diode (LED) light source, and a structured light illumination light field with light intensity meeting one-dimensional cosine function distribution is formed on a sample through the modulation of a Spatial Light Modulator (SLM) or a Digital Micromirror Device (DMD) light field modulation device; the intensity of the structured light illumination field satisfies the distribution of formula (1);
Figure FDA0002640180340000071
where r denotes the two-dimensional object coordinate of the sample plane, I0Representing the mean value of the intensity of the structured light field, m representing the degree of modulation, k0The spatial frequency is represented by a representation of,
Figure FDA0002640180340000072
representing an initial phase;
by utilizing the nonlinear response characteristic of the fluorescent molecules, the intensity of the generated fluorescent signal light has the optical field distribution described by the formula (7):
Figure FDA0002640180340000073
where r denotes the two-dimensional object coordinate of the sample plane, I0Representing the mean value of the light intensity, M the harmonic order of the structured light field, bnIndicating the degree of modulation, k, of the nth harmonic0Which represents the spatial frequency of the fundamental frequency,
Figure FDA0002640180340000074
representing an initial phase;
in the case of harmonic order M being 3, the light intensity distribution of the fluorescence signal generated by the nonlinear excitation response satisfies the structural light field described by formula (12):
Figure FDA0002640180340000081
controlling the light field modulation device to generate 7 structured light fields with different spatial directions and an included angle theta of 51.4 degrees between the adjacent directions on a sample plane; in each spatial direction, it is necessary to shift the phase by the amount
Figure FDA0002640180340000082
Generating 7 structured light fields with different phase shifts; so as to generate 49 light fields of the excited fluorescence structure at different positions;
step 2), collecting a fluorescence microscopic image by an area array digital camera:
after 7 phase shift structure light fields corresponding to 7 spatial directions are imaged by a fluorescence microscope system, 49 fluorescence images illuminated by structural light are sequentially recorded and obtained on an image surface of the imaging system by an area array digital camera CMOS or CCD, the fluorescence images are divided into 7 groups according to the spatial directions, and each group comprises 7 different phases which are respectively recorded as: { D11(r’)，D12(r’),……,D17(r’)}、{D21(r’)，D22(r’),……,D27(r’)}、……、{D71(r’)，D72(r’),……,D77(r ') }, where r' represents image plane two-dimensional image coordinates;
step 3), super-resolution image reconstruction processing:
step 3.1), according to the knotMean value of light intensity I of structured light field0Modulation b of each harmonic wave 30、b1、b2、b3Fundamental frequency k of space0Initial phase in 7 spatial directions
Figure FDA0002640180340000083
Figure FDA0002640180340000084
And phase shift amount 2 pi/7, obtaining a demodulation coefficient matrix described by formula (13), and correspondingly dividing the demodulation coefficient matrix into 7 groups according to the space direction, wherein each group contains 7 different phases which are respectively recorded as: { c11(r’)，c12(r’),……,c17(r’)}、{c21(r’)，c22(r’),……,c27(r’)}、……、{c71(r’)，c72(r’),……,c77(r’)}：
Figure FDA0002640180340000091
Step 3.2), calculating the 49 structural light illuminated fluorescence images acquired in the step 2) and the demodulation coefficient matrix generated in the step 3.1) according to the sequence by a formula (14) to obtain an initial super-resolution image RSDR(r’)：
Figure FDA0002640180340000092
Step 3.3), generating a new super-resolution equivalent point spread function P (r ') according to a formula (15) by using an optical system point spread function H (r') obtained theoretically or experimentally:
P(r’)=H(r’)×{b0+b2cos[2πk0r’]+b2cos[2(2πk0r’)]+b3cos[3(2πk0r’)]} (15)
step 3.4), using the initial super-resolution image R obtained in step 3.2)SDR(r') and the super-resolution equivalent point spread function generated in step 3.3)The number P (R') is counted, the deconvolution operation is completed, and the final super-resolution image R is obtainedSIM(r’)。
11. A fast system for directly reconstructing a structured light illumination super-resolution image in a space domain comprises a processor and a memory, and is characterized in that: the memory has stored therein a computer program which, when run on the processor, performs the method of step 3) of claim 10.
12. A computer-readable storage medium characterized by: a computer program is stored which, when executed, implements the method of step 3) of claim 10.
CN201911238624.6A 2019-12-06 2019-12-06 Rapid method and system for directly reconstructing structured light illumination super-resolution image in space domain Active CN111077121B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911238624.6A CN111077121B (en) 2019-12-06 2019-12-06 Rapid method and system for directly reconstructing structured light illumination super-resolution image in space domain

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911238624.6A CN111077121B (en) 2019-12-06 2019-12-06 Rapid method and system for directly reconstructing structured light illumination super-resolution image in space domain

Publications (2)

Publication Number Publication Date
CN111077121A CN111077121A (en) 2020-04-28
CN111077121B true CN111077121B (en) 2020-11-17

Family

ID=70313213

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911238624.6A Active CN111077121B (en) 2019-12-06 2019-12-06 Rapid method and system for directly reconstructing structured light illumination super-resolution image in space domain

Country Status (1)

Country Link
CN (1) CN111077121B (en)

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111458318B (en) * 2020-05-12 2021-06-22 西安交通大学 Super-resolution imaging method and system utilizing square lattice structure light illumination
CN111458317B (en) * 2020-05-12 2021-04-30 北京大学 Direct structured light illumination super-resolution microscopic reconstruction method
CN111948296A (en) * 2020-07-20 2020-11-17 南昌大学 Method for improving axial resolution of photoacoustic microscopic imaging system based on axial modulation
CN113112405B (en) * 2021-04-12 2022-04-12 广州超视计生物科技有限公司 Self-adaptive correction method of super-resolution microscope image and SIM-ODT (subscriber identity module-ODT) bimodal system
CN113466192A (en) * 2021-06-16 2021-10-01 浙大宁波理工学院 Ultra-high-speed real-time super-resolution microscopic imaging method
CN113670878B (en) * 2021-08-25 2022-09-27 西安交通大学 Super-resolution structured light illumination microscopic imaging method based on space-frequency domain hybrid reconstruction
CN113947565B (en) * 2021-09-03 2023-04-18 中国科学院西安光学精密机械研究所 Structured light illumination super-resolution imaging gene detection method based on deep learning
CN115839936B (en) * 2022-12-12 2023-07-18 之江实验室 Structured light illumination super-resolution microscopic imaging reconstruction method based on phase-locked detection

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101655601B (en) * 2008-08-22 2012-09-26 麦克奥迪实业集团有限公司 Light microscope imaging method and system based on DMD structure
CN102540446B (en) * 2011-12-28 2014-03-26 中国科学院西安光学精密机械研究所 High-speed structure illumination optical microscope system and method based on digital micromirror device
GB201318598D0 (en) * 2013-10-21 2013-12-04 Univ Leicester Improvements in or relating to super-resolution microscopy
CN103645136A (en) * 2013-11-22 2014-03-19 深圳先进技术研究院 Method and device for improving multiphoton fluorescence microscope imaging resolution
CN104515759B (en) * 2014-12-16 2017-05-31 中国科学院苏州生物医学工程技术研究所 Nonlinear organization optical illumination micro imaging method and system
CN106054380A (en) * 2016-07-15 2016-10-26 中国科学院苏州生物医学工程技术研究所 Light section fluorescent image lighting method through FPGA acceleration processing structured light
CN106770147B (en) * 2017-03-15 2019-07-19 北京大学 A kind of Structured Illumination super-resolution micro imaging method
CN107389631B (en) * 2017-04-28 2020-07-07 中国科学院生物物理研究所 High-speed multicolor multi-modal structured light illumination super-resolution microscopic imaging system and method thereof
CN107180411B (en) * 2017-05-19 2021-05-18 中国科学院苏州生物医学工程技术研究所 Image reconstruction method and system
CN107966826B (en) * 2017-12-27 2019-07-05 中国科学院半导体研究所 A kind of small-scale structure optical illumination super-resolution micro imaging system
JP2019144807A (en) * 2018-02-20 2019-08-29 浜松ホトニクス株式会社 Image processing method, image processing device, and image processing program

Also Published As

Publication number Publication date
CN111077121A (en) 2020-04-28

Similar Documents

Publication Publication Date Title
CN111077121B (en) Rapid method and system for directly reconstructing structured light illumination super-resolution image in space domain
CN107389631B (en) High-speed multicolor multi-modal structured light illumination super-resolution microscopic imaging system and method thereof
US8019136B2 (en) Optical sectioning microscopy
CN111458318B (en) Super-resolution imaging method and system utilizing square lattice structure light illumination
CN110954521B (en) Wide-field super-resolution microscopic imaging method and system thereof
CN105784653B (en) A kind of wide field super-resolution fluorescence microscopic imaging device
CN113670878B (en) Super-resolution structured light illumination microscopic imaging method based on space-frequency domain hybrid reconstruction
CN105929560A (en) Broadband far field super-resolution imaging apparatus
US11221297B2 (en) Accelerated methods and apparatuses for three-dimensional microscopy with structured illumination
Chen et al. Superresolution structured illumination microscopy reconstruction algorithms: a review
CN110954523B (en) Two-photon scanning structure light microscopic imaging method and device
US20220076379A1 (en) Method for super-resolution evaluation of microscope images illuminated in a structured manner and microscope having structured illumination
CN113484296A (en) Super-resolution scanning light field imaging system and method based on structured light illumination
JP5085608B2 (en) Wide-field super-resolution optical microscope using a spatial light modulator
Dan et al. Rapid image reconstruction of structured illumination microscopy directly in the spatial domain
CN114594588B (en) Structured light illumination microscopic device and method based on grating projection and SLM phase shift
CN114967092B (en) Super-high-speed structured light illumination super-resolution microscopic imaging device based on compressed sensing
Stemmer et al. Widefield fluorescence microscopy with extended resolution
Gong et al. GPU-accelerated real-time reconstruction in Python of three-dimensional datasets from structured illumination microscopy with hexagonal patterns
CN111024664B (en) Imaging method of structured light illumination super-resolution microscopic chip
JP2024016219A (en) Programmable multipoint illuminators, confocal filters, confocal microscopes, and how to operate confocal microscopes
US11947098B2 (en) Multi-focal light-sheet structured illumination fluorescence microscopy system
JP2017523439A (en) Double structured illumination with modulated phase and intensity
CN113568294B (en) Holographic optical tweezers fusion structure light illumination microscopic system and method
Enderlein Advanced fluorescence microscopy

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
TR01 Transfer of patent right
TR01 Transfer of patent right

Effective date of registration: 20210517

Address after: 518083 the comprehensive building of Beishan industrial zone and 11 2 buildings in Yantian District, Shenzhen, Guangdong.

Patentee after: Shenzhen Huada Zhizao Technology Co., Ltd

Address before: 710119, No. 17, information Avenue, new industrial park, hi tech Zone, Shaanxi, Xi'an

Patentee before: Xi'an Institute of Optics and precision machinery, Chinese Academy of Sciences