CN111077121A - Rapid method and system for directly reconstructing structured light illumination super-resolution image in space domain - Google Patents
Rapid method and system for directly reconstructing structured light illumination super-resolution image in space domain Download PDFInfo
- Publication number
- CN111077121A CN111077121A CN201911238624.6A CN201911238624A CN111077121A CN 111077121 A CN111077121 A CN 111077121A CN 201911238624 A CN201911238624 A CN 201911238624A CN 111077121 A CN111077121 A CN 111077121A
- Authority
- CN
- China
- Prior art keywords
- light
- super
- structured light
- resolution image
- spatial
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/62—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light
- G01N21/63—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light optically excited
- G01N21/64—Fluorescence; Phosphorescence
- G01N21/645—Specially adapted constructive features of fluorimeters
- G01N21/6456—Spatial resolved fluorescence measurements; Imaging
- G01N21/6458—Fluorescence microscopy
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B21/00—Microscopes
- G02B21/0004—Microscopes specially adapted for specific applications
- G02B21/002—Scanning microscopes
- G02B21/0024—Confocal scanning microscopes (CSOMs) or confocal "macroscopes"; Accessories which are not restricted to use with CSOMs, e.g. sample holders
- G02B21/0052—Optical details of the image generation
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B21/00—Microscopes
- G02B21/0004—Microscopes specially adapted for specific applications
- G02B21/002—Scanning microscopes
- G02B21/0024—Confocal scanning microscopes (CSOMs) or confocal "macroscopes"; Accessories which are not restricted to use with CSOMs, e.g. sample holders
- G02B21/0052—Optical details of the image generation
- G02B21/0076—Optical details of the image generation arrangements using fluorescence or luminescence
Abstract
The invention relates to a rapid method and a rapid system for directly reconstructing a structured light illumination super-resolution image in a space domain, which can be widely applied to the research in the fields of biology, medicine, material science, microelectronics and the like. Compared with the traditional frequency domain SIM super-resolution image reconstruction method, the method disclosed by the invention has the advantages that the operation is completely performed in a space domain, the processes of frequency spectrum separation, frequency spectrum movement, frequency spectrum fusion and the like are avoided, the time-consuming multiple forward/inverse Fourier transform operation is avoided, and the problem of artifact noise caused by frequency spectrum operation can be completely avoided; the demodulation matrix for super-resolution reconstruction can be generated in advance, the related operations are only multiplication and addition operations, and the process is simple and easy to implement, so that the reconstruction speed of the SIM super-resolution image is greatly improved.
Description
Technical Field
The invention relates to a rapid method for directly reconstructing a structured light illumination super-resolution image in a space domain, which can be widely applied to the research in the fields of biology, medicine, material science, microelectronics and the like.
Background
The spatial resolution of conventional optical microscopes is limited by the diffraction limit of light, which can only reach half the wavelength of light, and greatly limits the application range of optical microscopes, for example, in imaging structures in living biological cells. Achieving high spatial resolution imaging has been one of the important research topics in the field of optical microscopy. By labeling the sample with fluorescent molecules, the imaging signal-to-noise ratio and contrast of the optical microscopy technique are significantly improved. Based on the intensity response mechanism of fluorescent molecules to illumination light, various super-resolution fluorescence Microscopy technologies are proposed, such as Photo-activated localization Microscopy (PALM), random optical reconstruction fluorescence Microscopy (STORM), stimulated emission Depletion Microscopy (STED), structured light illumination fluorescence Microscopy (SIM), and the like. The highest resolution of the current super-resolution fluorescence microscopic imaging technology is close to the resolution level of an electron microscope, provides a powerful tool for modern biomedicine, and also pushes related research to a new depth. In the super-resolution fluorescence microscopy technology, the SIM has the advantages of high imaging speed, no special requirement on fluorescent molecules, photobleaching, low phototoxicity and the like, and is particularly suitable for long-time dynamic observation of living cells under high resolution. SIMs can be classified into linear SIMs and non-linear SIMs based on the linear and non-linear responses of fluorescence to illumination light. The linear SIM adopts a structural light field with cosine distribution of intensity to illuminate the sample, the distribution of the excited fluorescence light field is in linear relation with the illumination light field, and the spatial resolution exceeds twice of the diffraction limit. Since the spatial resolution of the microscope system depends on the maximum spatial frequency f it can collect0And f is0Depending on the optical transfer function OTF of the system. When the sample contains high frequenciesInformation f>f0The details of the sample cannot be distinguished. If a spatial frequency f is used1The cosine fringe structure light illuminates the sample to generate a spatial frequency fm=|f-f1Low frequency moire of l. Moire fringes are actually beat frequency signals of sample and structure light fields, and contain high frequency information f of sample super-diffraction resolution. Due to fm<f0The moire fringes can be recorded by a microscope system, and then the high-frequency information f of the sample can be extracted through algorithm demodulation, so that a high-resolution image of the sample is reconstructed. The nonlinear SIM is developed on the basis of the theory of the linear SIM, and by utilizing the nonlinear response characteristic of fluorescent molecules to the illumination light intensity, a fluorescence light field generated by excitation shows trapezoidal wave or square wave distribution with higher-order frequency, the formed moire fringe contains more high-frequency components of a sample, and the details of the finally reconstructed sample are richer. The super-resolution image reconstruction method is the core technology of the SIM. The current image reconstruction method processes data mainly in the spatial frequency domain (abbreviated as FDR method), and its disadvantages mainly lie in: firstly, Fourier transform, spectrum separation, spectrum movement, spectrum fusion, inverse Fourier transform and other processes are needed, and the steps are various; secondly, the reconstruction is very time-consuming due to multiple forward/inverse fourier transform operations. In order to reconstruct a super-resolution image, a linear SIM generally needs to acquire 6 or 9 structured light illumination images for operation, while a non-linear SIM needs more. As image size increases, reconstruction time consumption becomes more significant. Finally, operating (separating, moving, fusing) on the frequency components of the image data is easy to bring artifact noise, and the real details of the sample are covered. Due to the defects of the traditional super-resolution image reconstruction method, the current SIM system is difficult to realize real-time super-resolution dynamic observation of samples, and the application of the SIM in living subcellular dynamic observation is hindered.
Disclosure of Invention
Aiming at the problem that the current structured light illumination super-resolution image reconstruction method is difficult in super-resolution microscopy real-time dynamic observation due to complex operation and serious time consumption, the invention provides a rapid method (SDR method for short) for directly reconstructing a structured light illumination super-resolution image in a space domain, the operation steps are greatly simplified compared with the traditional method, the steps are reduced from 5 traditional steps to 2 steps, and basic multiplication and addition operations are carried out without Fourier transform operation, so the reconstruction speed is greatly improved.
The technical scheme of the invention is to provide a rapid method for directly reconstructing a structured light illumination super-resolution image in a space domain, which comprises the following steps under the condition of linear excitation response:
step 1), generation and phase shift of a structured light field:
the light is illuminated by a laser or a Light Emitting Diode (LED) light source and modulated by a Spatial Light Modulator (SLM) or a Digital Micromirror Device (DMD) light field modulator, and a structured light illumination light field with light intensity meeting one-dimensional cosine function distribution is formed on a sample; the intensity of the structured light illumination field satisfies the distribution of formula (1):
where r denotes the two-dimensional object coordinate of the sample plane, I0Representing the mean value of the intensity of the structured light field, m representing the degree of modulation, k0The spatial frequency is represented by a representation of,representing an initial phase;
controlling the light field modulation device to generate 3 structural light fields with different spatial directions and an included angle theta of the adjacent directions being 1200 in a sample plane; in each spatial direction by the amount of phase shiftGenerating 3 structured light fields with different phase shifts, the phases of which are respectivelyAndso as to generate 9 structured light fields at different positions for illuminating and exciting the sampleThe product generates a fluorescent signal;
step 2), collecting a fluorescence image by an area array digital camera:
the method comprises the following steps that (1) an illumination light field with 3 phase shift structures corresponding to 3 spatial directions is imaged by a fluorescence microscope system, 9 fluorescence images illuminated by structural light are sequentially obtained by recording by an area array digital camera CMOS or CCD on an image surface of the imaging system, the fluorescence images are divided into 3 groups according to the spatial directions, each group comprises 3 different phases and are respectively recorded as: { D11(r’),D12(r’),D13(r’)}、{D21(r’),D22(r’),D23(r’)}、{D31(r’),D32(r’),D33(r ') }, where r' represents image plane two-dimensional image coordinates;
step 3), super-resolution image reconstruction processing:
step 3.1), according to the light intensity average value I of the structured light field0Degree of modulation m, spatial frequency k0Initial phase in 3 spatial directionsAnd the phase shift amount is 2 pi/3, a demodulation coefficient matrix shown in the formula (2) is generated, the demodulation coefficient matrix is also correspondingly divided into 3 groups according to the space direction, each group contains 3 different phases which are respectively recorded as: { c11(r’),c12(r’),c13(r’)}、{c21(r’),c22(r’),c23(r’)}、{c31(r’),c32(r’),c33(r’)};
Step 3.2), calculating the 9 structural light illuminated fluorescence images collected in the step 2) and the demodulation coefficient matrix generated in the step 3.1) according to the sequence by the formula (3) to obtain an initial super-resolution image RSDR(r’):
Step 3.3), generating a new super-resolution equivalent point spread function P (r ') according to a formula (4) by using an optical system point spread function H (r') obtained theoretically or experimentally:
P(r′)=H(r′)[1+cos(2πk0r′)](4)
step 3.4), using the initial super-resolution image R obtained in step 3.2)SDR(R ') and the super-resolution equivalent point spread function P (R') generated in the step 3.3), and the deconvolution operation is completed to obtain the final super-resolution image RSIM(r’)。
The invention also provides a rapid system for directly reconstructing the structured light illumination super-resolution image in the airspace, which comprises a processor and a memory and is characterized in that: the memory stores a computer program which, when executed on the processor, performs the method of step 3) above.
The invention also provides a computer-readable storage medium, which is characterized in that: a computer program is stored which, when executed, implements the method described above in step 3).
The invention also provides another rapid method for directly reconstructing the structured light illumination super-resolution image in the space domain, which comprises the following steps under the condition of linear excitation response:
step 1), generation and phase shift of a structured light field:
the light is illuminated by a laser or a Light Emitting Diode (LED) light source, and a structured light illumination light field with light intensity meeting one-dimensional cosine function distribution is formed on a sample through the modulation of a Spatial Light Modulator (SLM) or a Digital Micromirror Device (DMD) light field modulation device; the intensity of the structured light illumination field satisfies the distribution of formula (1):
where r denotes the two-dimensional object coordinate of the sample plane, I0Representing the mean value of the intensity of the structured light field, m representing the degree of modulation, k0The spatial frequency is represented by a representation of,indicating the initial phase;
Controlling the light field modulation device to generate 2 structured light fields with different spatial directions and an included angle theta of 90 degrees between the adjacent directions in a sample plane; in each spatial direction by the amount of phase shiftGenerating 3 structured light fields with different phase shifts, the phases of which are respectivelyAndgenerating structural light fields at 6 different positions in sequence for illuminating and exciting the sample to generate a fluorescence signal;
step 2), collecting a fluorescence image by an area array digital camera:
the method comprises the following steps that (1) an illumination light field with 3 phase shift structures corresponding to 2 spatial directions is imaged by a fluorescence microscope system, 6 fluorescence images illuminated by structural light are sequentially recorded on an image surface of the imaging system by an area array digital camera CMOS or CCD, the fluorescence images are divided into 2 groups according to the spatial directions, each group comprises 3 different phases and are respectively recorded as: { D11(r’),D12(r’),D13(r’)}、{D21(r’),D22(r’),D23(r’)};
Step 3), super-resolution image reconstruction processing:
step 3.1), according to the light intensity average value I of the structured light field0Degree of modulation m, spatial frequency k 02 initial phase in spatial directionAnd phase shift amount pi/2, generating a demodulation coefficient matrix shown in formula (5), and correspondingly dividing the demodulation coefficient matrix into 2 groups according to the spatial direction, wherein each group contains 3 different phases which are respectively recorded as: { c11(r’),c12(r’),c13(r’)}、{c21(r’),c22(r’),c23(r’)};
Step 3.2), calculating the 6 structural light illuminated fluorescence images collected in the step 2) and the demodulation coefficient matrix generated in the step 3.1) according to the sequence by a formula (6) to obtain an initial super-resolution image RSDR(r’):
Step 3.3), generating a new super-resolution equivalent point spread function P (r ') according to a formula (4) by using an optical system point spread function H (r') obtained theoretically or experimentally;
P(r′)=H(r′)[1+cos(2πk0r′)](4)
step 3.4), using the initial super-resolution image R obtained in step 3.2)SDR(R ') and the super-resolution equivalent point spread function P (R') generated in the step 3.3), and the deconvolution operation is completed to obtain the final super-resolution image RSIM(r’)。
The invention also provides a rapid system for directly reconstructing the structured light illumination super-resolution image in the airspace, which comprises a processor and a memory and is characterized in that: the memory stores a computer program which, when executed on the processor, performs the method of step 3) above.
The invention also provides a computer-readable storage medium, which is characterized in that: a computer program is stored which, when executed, implements the method described above in step 3).
The invention also provides another rapid method for directly reconstructing the structured light illumination super-resolution image in the space domain, which comprises the following steps under the condition of nonlinear excitation response:
step 1), excitation and phase shift of a nonlinear structure light field:
the light is illuminated by a laser or a Light Emitting Diode (LED) light source, and a structured light illumination light field with light intensity meeting one-dimensional cosine function distribution is formed on a sample through the modulation of a Spatial Light Modulator (SLM) or a Digital Micromirror Device (DMD) light field modulation device; the intensity of the structured light illumination field satisfies the distribution of formula (1);
where r denotes the two-dimensional object coordinate of the sample plane, I0Representing the mean value of the intensity of the structured light field, m representing the degree of modulation, k0The spatial frequency is represented by a representation of,representing an initial phase;
by utilizing the nonlinear response characteristic of the fluorescent molecules, the intensity of the generated fluorescent signal light has the optical field distribution described by the formula (7):
where r denotes the two-dimensional object coordinate of the sample plane, I0Representing the mean value of the light intensity, M the harmonic order of the structured light field, bnIndicating the degree of modulation, k, of the nth harmonic0Which represents the spatial frequency of the fundamental frequency,representing an initial phase;
controlling the optical field modulation device to generate (2M +1) structural optical fields with different spatial directions and an included angle theta between adjacent directions being 360 degrees/2M +1 on a sample plane; in each spatial direction, it is necessary to shift the phase by the amountGenerating (2M +1) structured light fields with different phase shifts; in the case of harmonic order M being 2, the light intensity distribution of the fluorescence signal generated by the nonlinear excitation response satisfies the structural light field described by formula (8):
in this case, it is necessary to generate on the sample plane5 structured light fields with different spatial directions and an included angle theta between the adjacent directions being 720; in each spatial direction, it is necessary to shift the phase by the amountGenerating 5 structured light fields with different phase shifts; so as to generate 25 excited fluorescence structure light fields at different positions;
2) acquiring a fluorescence image by an area array digital camera:
after imaging by a fluorescence microscope system, 25 fluorescence images illuminated by structural light are sequentially obtained on an image plane of the imaging system by an area array digital camera CMOS or CCD, the fluorescence images are divided into 5 groups according to the spatial direction, each group contains 5 different phases and are respectively marked as: { D11(r’),D12(r’),……,D15(r’)}、{D21(r’),D22(r’),……,D25(r’)}、……、{D51(r’),D52(r’),……,D55(r ') }, where r' represents image plane two-dimensional image coordinates;
step 3), super-resolution image reconstruction processing:
step 3.1), according to the light intensity average value I of the structured light field0Modulation b of each harmonic wave, M20、b1、b2Fundamental frequency k of space0Initial phase in 5 spatial directionsAnd phase shift amount 2 pi/5, obtaining a demodulation coefficient matrix described by formula (9), and correspondingly dividing the demodulation coefficient matrix into 5 groups according to the spatial direction, wherein each group contains 5 different phases which are respectively recorded as: { c11(r’),c12(r’),……,c15(r’)}、{c21(r’),c22(r’),……,c25(r’)}、……、{c51(r’),c52(r’),……,c55(r’)}:
Wherein i ═ 1,2,3,4,5.(9)
Step 3.2), calculating the 25 structured light illuminated fluorescence images acquired in the step 2) and the demodulation coefficient matrix generated in the step 3.1) according to the sequence through a formula (10) to obtain an initial super-resolution image RSDR(r’):
Step 3.3), generating a new super-resolution equivalent point spread function P (r ') according to a formula (11) by using an optical system point spread function H (r') obtained theoretically or experimentally:
P(r′)=H(r′)×{b0+b1cos[2πk0r′]+b2cos[2(2πk0r′)]} (11)
step 3.4), using the initial super-resolution image R obtained in step 3.2)SDR(R ') and the super-resolution equivalent point spread function P (R') generated in the step 3.3), and the deconvolution operation is completed to obtain the final super-resolution image RSIM(r’)。
The invention also provides a rapid system for directly reconstructing the structured light illumination super-resolution image in the airspace, which comprises a processor and a memory and is characterized in that: the memory stores a computer program which, when executed on the processor, performs the method of step 3) above.
The invention also provides a computer-readable storage medium, which is characterized in that: a computer program is stored which, when executed, implements the method described above in step 3).
The invention also provides another rapid method for directly reconstructing the structured light illumination super-resolution image in the space domain, which comprises the following steps under the condition of nonlinear excitation response:
step 1), excitation and phase shift of a nonlinear structure light field:
the light is illuminated by a laser or a Light Emitting Diode (LED) light source, and a structured light illumination light field with light intensity meeting one-dimensional cosine function distribution is formed on a sample through the modulation of a Spatial Light Modulator (SLM) or a Digital Micromirror Device (DMD) light field modulation device; the intensity of the structured light illumination field satisfies the distribution of formula (1);
where r denotes the two-dimensional object coordinate of the sample plane, I0Representing the mean value of the intensity of the structured light field, m representing the degree of modulation, k0The spatial frequency is represented by a representation of,representing an initial phase;
by utilizing the nonlinear response characteristic of the fluorescent molecules, the intensity of the generated fluorescent signal light has the optical field distribution described by the formula (7):
where r denotes the two-dimensional object coordinate of the sample plane, I0Representing the mean value of the light intensity, M the harmonic order of the structured light field, bnIndicating the degree of modulation, k, of the nth harmonic0Which represents the spatial frequency of the fundamental frequency,representing an initial phase;
in the case of harmonic order M being 3, the light intensity distribution of the fluorescence signal generated by the nonlinear excitation response satisfies the structural light field described by formula (12):
controlling the light field modulation device to generate 7 different spatial directions on the sample plane, wherein the included angle theta between the adjacent directions is 51.40The structured light field of (1); in each spatial direction, it is necessary to shift the phase by the amountGenerating 7 structured light fields with different phase shifts; so as to generate 49 pieces ofCo-located light field exciting the fluorescent structure;
step 2), collecting a fluorescence microscopic image by an area array digital camera:
after 7 phase shift structure light fields corresponding to 7 spatial directions are imaged by a fluorescence microscope system, 49 fluorescence images illuminated by structural light are sequentially recorded and obtained on an image surface of the imaging system by an area array digital camera CMOS or CCD, the fluorescence images are divided into 7 groups according to the spatial directions, and each group comprises 7 different phases which are respectively recorded as: { D11(r’),D12(r’),……,D17(r’)}、{D21(r’),D22(r’),……,D27(r’)}、……、{D71(r’),D72(r’),……,D77(r ') }, where r' represents image plane two-dimensional image coordinates;
step 3), super-resolution image reconstruction processing:
step 3.1), according to the light intensity average value I of the structured light field0Modulation b of each harmonic wave 30、b1、b2、b3Fundamental frequency k of space0Initial phase in 7 spatial directions And phase shift amount 2 pi/7, obtaining a demodulation coefficient matrix described by formula (13), and correspondingly dividing the demodulation coefficient matrix into 7 groups according to the space direction, wherein each group contains 7 different phases which are respectively recorded as: { c11(r’),c12(r’),……,c17(r’)}、{c21(r’),c22(r’),……,c27(r’)}、……、{c71(r’),c72(r’),……,c77(r’)}:
Wherein i is 1,2,3,4,5,6,7.(13)
Step 3.2), and step 3) of illuminating the 49 structured light fluorescence images acquired in the step 2).1) The demodulation coefficient matrix generated in (1) is calculated by formula (14) in sequence to obtain an initial super-resolution image RSDR(r’):
Step 3.3), generating a new super-resolution equivalent point spread function P (r ') according to a formula (15) by using an optical system point spread function H (r') obtained theoretically or experimentally:
P(r′)=H(r′)×{b0+b1cos[2πk0r′]+b2cos[2(2πk0r′)]+b3cos[3(2πk0r′)]} (15)
step 3.4), using the initial super-resolution image R obtained in step 3.2)SDR(R ') and the super-resolution equivalent point spread function P (R') generated in the step 3.3), and the deconvolution operation is completed to obtain the final super-resolution image RSIM(r’)。
The invention also provides a rapid system for directly reconstructing the structured light illumination super-resolution image in the airspace, which comprises a processor and a memory and is characterized in that: the memory stores a computer program which, when executed on the processor, performs the method of step 3) above.
The invention also provides a computer-readable storage medium, which is characterized in that: a computer program is stored which, when executed, implements the method described above in step 3).
The invention has the advantages that:
the invention directly and rapidly reconstructs the SIM super-resolution image in a space domain. Compared with the traditional frequency domain SIM super-resolution image reconstruction method, the method disclosed by the invention has the advantages that the operation is completely carried out in a space domain, the processes of frequency spectrum separation, frequency spectrum movement, frequency spectrum fusion and the like are avoided, the time-consuming multiple forward/inverse Fourier transform operations are avoided, and the problem of artifact noise caused by frequency spectrum operation can be completely avoided; the demodulation matrix for super-resolution reconstruction can be generated in advance, the related operations are only multiplication and addition operations, and the process is simple and easy to implement, so that the reconstruction speed of the SIM super-resolution image is greatly improved.
Drawings
FIG. 1 illustrates the operation steps and comparison of the present invention method and conventional method for super-resolution image reconstruction;
FIGS. 2a and 2b are graphs showing the calculation time consumption statistics and comparison of super-resolution image reconstruction by the method of the present invention and the conventional method in different image sizes;
FIG. 3 is a light path diagram of a projection type SIM super-resolution microscope system based on digital micromirror DMD modulation and LED illumination;
the reference numbers in the figures are: 1-LED illumination light source, 2-beam splitting prism, 3-structured light generator, 4-collimating lens, 5-illumination light filter, 6-beam splitter, 7-reflector, 8-microscope objective, 9-objective stage, 10-filter, 11-tube lens and 12-area array digital camera;
FIG. 4 is a light path diagram of an interferometric SIM super-resolution microscope system based on spatial light modulator SLM modulation and zero-order vortex half-wave plate polarization control;
the reference numbers in the figures are: the system comprises a 1-laser illumination source, 2,3, 7, 10, 11-lenses, a 4-polarization beam splitter, a 5-half wave plate, a 6-spatial light modulator SLM, an 8-spatial filter, a 9-zero-order vortex half wave plate, a 12-beam splitter, a 13-microscope objective, a 14-objective table, a 15-optical filter, a 16-tube lens and a 17-area array digital camera.
Fig. 5 is a three-channel super-resolution microscopic image of bovine pulmonary artery endothelial cells (BPAE) obtained using a DMD modulation and LED illumination based SIM super-resolution microscopic system. The blue channel is a cell nucleus image under the excitation of 405nm light, the green channel is an actin image under the excitation of 470nm light, and the red channel is a mitochondria image under the excitation of 565nm light; wherein (a) is a normal wide-field fluorescence image of the sample, (b) is a structured light illuminated three-channel super-resolution microscopy image reconstructed using a conventional method, (c) is a structured light illuminated three-channel super-resolution microscopy image reconstructed using the method of the present invention, (d) - (f) are enlarged views of the dashed box regions in (a) - (c);
FIG. 6 is a super-resolution microscopic image of mitochondria in bovine pulmonary artery endothelial cells obtained by using a Spatial Light Modulator (SLM) -based modulation and a zero-order vortex half-wave plate polarization control SIM super-resolution microscopic system. Wherein (a) is a normal wide-field fluorescence image, (b) is a structured light illuminated super-resolution microscopy image reconstructed using a conventional method, (c) is a structured light illuminated super-resolution microscopy image reconstructed using the method of the present invention, (d) - (f) are enlarged views of the dashed box regions in (a) - (c);
fig. 7 is a graph of the intensity distribution along the line marked in fig. 6(d) -6 (f).
Detailed Description
The invention is further described with reference to the following figures and specific embodiments.
The method can be implemented on a mainstream SIM super-resolution microscope system, and comprises a projection type SIM super-resolution microscope system based on digital micromirror DMD modulation and LED illumination, an interference type SIM super-resolution microscope system based on spatial light modulator SLM modulation and zero-order vortex half-wave plate polarization control, and the like. Compared with the traditional method, the operation steps are greatly simplified, as shown in the comparison of fig. 1, the traditional steps comprise 5 steps of Fourier transform, frequency spectrum separation, frequency spectrum movement, frequency spectrum fusion and inverse Fourier transform, while the invention reduces the traditional 5 steps into 2 steps, and the basic multiplication and addition operations are carried out, so that the Fourier transform operation is not needed, and the reconstruction speed is greatly improved.
A projection type SIM super-resolution microscope system based on digital micromirror DMD modulation and LED illumination is shown in fig. 3: the device comprises an LED illumination light source 1, a total internal reflection beam splitter prism 2 arranged on an illumination light path, a structured light generator 3 arranged on a reflection light path of the beam splitter prism 2, a collimating lens 4 and an illumination light filter 5 which are sequentially arranged on a transmission light path of the beam splitter prism 2, a beam splitter 6 arranged behind the illumination light filter 5, a reflector 7 arranged on a transmission light path of the beam splitter 6, a microscope objective 8 and an objective table 9 which are arranged on a light path behind the reflector 7, a light filter 10 and a barrel lens 11 which are sequentially arranged on a fluorescence reflection light path of the beam splitter 6, and an area array digital camera 12 arranged behind the barrel lens 11. The LED illumination light source 1 is an incoherent four-color LED light source, and the structured light generator 3 is a digital micromirror DMD.
An interferometric SIM super-resolution microscope system based on spatial light modulator SLM modulation and zero-order vortex half-wave plate polarization control is shown in figure 4: comprises a laser illumination light source 1, a beam expanding and collimating lens group (a lens 2 and a lens 3) arranged at the rear end of the laser illumination light source 1, a polarized beam splitter 4 arranged behind the beam expanding and collimating lens group, a half-wave plate 5 and a spatial light modulator SLM6 which are sequentially arranged on a transmission light path of the polarized beam splitter 4, a lens 7 arranged on a reflection light path of the polarized beam splitter 4, and a spatial filter 8 arranged at the rear end of the lens 7, the device comprises a zero-order vortex half-wave plate 9 arranged behind a spatial filter 8, a 4f system consisting of a lens 10 and a lens 11 arranged behind the zero-order vortex half-wave plate 9, a spectroscope 12 arranged behind the 4f system, a microscope objective 13 and an objective table 14 arranged on a transmission light path of the spectroscope 12, an optical filter 15 and a barrel mirror 16 arranged on a fluorescence reflection light path of the spectroscope 12, and an area array digital camera 17 arranged behind the barrel mirror 16. The spatial light modulator SLM6 is a reflective ferroelectric liquid crystal spatial light modulator.
Example one
The present embodiment is based on the super-resolution image reconstruction of the projection SIM with digital micromirror DMD modulation and LED illumination, and is implemented by the following steps:
Secondly, the acquired 6 images and the generated demodulation coefficient matrix are calculated by a formula (6) in sequence to obtain an initial super-resolution image RSDR(r’):
Then, generating a new super-resolution equivalent point spread function P (r ') according to a formula (4) by using an optical system point spread function H (r') obtained theoretically or experimentally;
P(r′)=H(r′)[1+cos(2πk0r′)](4)
finally, the obtained initial super-resolution image R is utilizedSDR(R ') and the generated super-resolution equivalent point spread function P (R') to complete the deconvolution operation and obtain the final super-resolution image RSIM(r’)。
and 5, carrying out channel fusion on the obtained cell nucleus, actin and mitochondria super-resolution microscopic images to obtain a three-channel super-resolution microscopic image of the bovine pulmonary artery endothelial cell.
Fig. 5 is a three-channel super-resolution microscopic image of bovine pulmonary artery endothelial cells obtained using a projection SIM super-resolution microscopy system based on digital micromirror DMD modulation and LED illumination. The experiment used a 100X microscope objective with a numerical aperture NA of 1.49. (a) The method comprises the following steps of (a) obtaining a common wide-field fluorescence microscopic image of a bovine pulmonary artery endothelial cell sample, (b) obtaining a structured light illuminated three-channel super-resolution microscopic image reconstructed by using a traditional method, and (c) obtaining the structured light illuminated three-channel super-resolution microscopic image reconstructed by using the method. (d) - (f) is an enlarged view of the dashed-line frame region in (a) - (c). It can be seen that the quality of the images reconstructed by both methods is substantially the same. The reconstruction time of the super-resolution image of the method is 0.0447 seconds, while the reconstruction time of the traditional method is 3.747 seconds, and the reconstruction speed of the super-resolution image of the method is 84 times that of the traditional method.
The embodiment also provides a rapid system for directly reconstructing the structured light illumination super-resolution image in the spatial domain, which comprises a processor and a memory, wherein the memory stores a computer program, and the computer program executes the super-resolution image reconstruction processing method when running in the processor.
The present embodiment also provides a computer-readable storage medium storing a program that, when executed, implements the above-described super-resolution image reconstruction processing method. In some possible embodiments, the invention may also be implemented in the form of a program product comprising program code means for causing a terminal device to carry out the steps according to various exemplary embodiments of the invention described in the method part of the description above, when said program product is run on the terminal device.
A program product for implementing the above method, which may employ a portable compact disc read only memory (CD-ROM) and include program code, may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in the present invention, the computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
Example two
The embodiment is based on spatial light modulator SLM modulation and zero-order vortex half-wave plate polarization control, and the super-resolution image reconstruction of the interferometric SIM:
and 3, controlling the SLM to load 9 structured light fields with the included angles of 3 spatial directions being 1200 and the phases of each spatial direction being 0, 2 pi/3 and 4 pi/3 in sequence, and correspondingly acquiring 9 images by the area-array digital camera respectively, wherein the value is (D)11,D12,D13)、(D21,D22,D23) And (D)31,D32,D33) And stored in the computer.
Secondly, calculating the 9 structured light illuminated fluorescence images collected in the step 3) and the generated demodulation coefficient matrix by a formula (3) in sequence to obtain an initial super-resolution image RSDR(r’):
Then, a new super-resolution equivalent point spread function P (r ') is generated according to formula (4) from the optical system point spread function H (r') obtained theoretically or experimentally:
P(r′)=H(r′)[1+cos(2πk0r′)](4)
finally, the obtained initial super-resolution image R is utilizedSDRAnd (r ') and the generated super-resolution equivalent point spread function P (r') to complete deconvolution operation and obtain a mitochondrial super-resolution image of the bovine pulmonary artery endothelial cell.
FIG. 6 is a super-resolution image of mitochondria within bovine pulmonary artery endothelial cells obtained using an interferometric SIM super-resolution microscopy system based on Spatial Light Modulator (SLM) modulation and zero-order vortex half-waveplate polarization control. The experiment used a 100X microscope objective with a numerical aperture NA of 1.49. (a) Is a common wide-field fluorescence image, (b) is a structured light illumination super-resolution image reconstructed by using a traditional method, (c) is a structured light illumination super-resolution image reconstructed by using the method of the invention, (d) - (f) are enlarged images of dotted line frame areas in (a) - (c); fig. 7 is a graph of the intensity distribution along the line marked in fig. 6(d) -6 (f). It can be seen that the quality of the images reconstructed by both methods is substantially the same. The reconstruction time of the super-resolution image of the method is 0.0934 seconds, while the reconstruction time of the traditional method is 4.7842 seconds, and the reconstruction speed of the super-resolution image of the method is 51 times that of the traditional method.
The embodiment also provides a rapid system for directly reconstructing the structured light illumination super-resolution image in the spatial domain, which comprises a processor and a memory, wherein the memory stores a computer program, and the computer program executes the super-resolution image reconstruction processing method when running in the processor.
The present embodiment also provides a computer-readable storage medium storing a program that, when executed, implements the above-described super-resolution image reconstruction processing method. In some possible embodiments, the invention may also be implemented in the form of a program product comprising program code means for causing a terminal device to carry out the steps according to various exemplary embodiments of the invention described in the method part of the description above, when said program product is run on the terminal device.
A program product for implementing the above method, which may employ a portable compact disc read only memory (CD-ROM) and include program code, may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in the present invention, the computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
In order to show the superiority of the method in the reconstruction time of the SIM super-resolution image compared with the traditional method, under the condition that the sizes of 4 common images are 512 multiplied by 512, 1024 multiplied by 1024, 1600 multiplied by 1600 and 2048 multiplied by 2048 pixels, the reconstruction time of the super-resolution image of 200 groups of SIM original images successively uses the method and the traditional method is counted. Each set of SIM original images consists of 9 structured light illumination images. As shown in FIGS. 2a and 2b, the mean value of the reconstruction time of the method is dozens of times less than that of the conventional method in any image size, and the statistical variance is one order of magnitude less, thus showing the rapidity and stability of the method. With the increase of the image size, the ratio of the reconstruction time average value of the method of the invention to the conventional method is gradually increased, and reaches 51.4 under 2048 × 2048 pixels, which shows that the reconstruction speed of the method of the invention is more than 50 times that of the conventional method. The software for calculation is Matlab R2013a, and the computer is configured by an Intel i7-4790K CPU, a 32GB memory and a Windows 7x64(SP1) operating system.
Claims (12)
1. A rapid method for directly reconstructing a structured light illumination super-resolution image in a space domain is characterized by comprising the following steps of:
step 1), generation and phase shift of a structured light field:
the light is illuminated by a laser or a Light Emitting Diode (LED) light source and modulated by a Spatial Light Modulator (SLM) or a Digital Micromirror Device (DMD) light field modulator, and a structured light illumination light field with light intensity meeting one-dimensional cosine function distribution is formed on a sample; the intensity of the structured light illumination field satisfies the distribution of formula (1):
where r denotes the two-dimensional object coordinate of the sample plane, I0Representing the mean value of the intensity of the structured light field, m representing the degree of modulation, k0The spatial frequency is represented by a representation of,representing an initial phase;
controlling the light field modulation device to generate 3 structural light fields with different spatial directions and an included angle theta of 120 degrees between the adjacent directions in a sample plane; in each spatial direction by the amount of phase shiftGenerating 3 structured light fields with different phase shifts, the phases of which are respectivelyAndgenerating 9 structured light fields at different positions in sequence for illuminating and exciting the sample to generate a fluorescence signal;
step 2), collecting a fluorescence image by an area array digital camera:
the method comprises the following steps that (1) an illumination light field with 3 phase shift structures corresponding to 3 spatial directions is imaged by a fluorescence microscope system, 9 fluorescence images illuminated by structural light are sequentially obtained by recording by an area array digital camera CMOS or CCD on an image surface of the imaging system, the fluorescence images are divided into 3 groups according to the spatial directions, each group comprises 3 different phases and are respectively recorded as: { D11(r’),D12(r’),D13(r’)}、{D21(r’),D22(r’),D23(r’)}、{D31(r’),D32(r’),D33(r ') }, where r' represents image plane two-dimensional image coordinates;
step 3), super-resolution image reconstruction processing:
step 3.1), according to the light intensity average value I of the structured light field0Degree of modulation m, spatial frequency k0Initial phase in 3 spatial directionsAnd the phase shift amount is 2 pi/3, a demodulation coefficient matrix shown in the formula (2) is generated, the demodulation coefficient matrix is also correspondingly divided into 3 groups according to the space direction, each group contains 3 different phases which are respectively recorded as: { c11(r’),c12(r’),c13(r’)}、{c21(r’),c22(r’),c23(r’)}、{c31(r’),c32(r’),c33(r’)};
Step 3.2), calculating the 9 structural light illuminated fluorescence images collected in the step 2) and the demodulation coefficient matrix generated in the step 3.1) according to the sequence by the formula (3) to obtain an initial super-resolution image RSDR(r’):
Step 3.3), generating a new super-resolution equivalent point spread function P (r ') according to a formula (4) by using an optical system point spread function H (r') obtained theoretically or experimentally:
P(r′)=H(r′)[1+cos(2πk0r′)](4)
step 3.4), using the initial super-resolution image R obtained in step 3.2)SDR(R ') and the super-resolution equivalent point spread function P (R') generated in the step 3.3), and the deconvolution operation is completed to obtain the final super-resolution image RSIM(r’)。
2. A fast system for directly reconstructing a structured light illumination super-resolution image in a space domain comprises a processor and a memory, and is characterized in that: the memory has stored therein a computer program which, when run on the processor, performs the method of step 3) of claim 1.
3. A computer-readable storage medium characterized by: a computer program is stored which, when executed, implements the method of claim 1, step 3).
4. A rapid method for directly reconstructing a structured light illumination super-resolution image in a space domain is characterized by comprising the following steps of:
step 1), generation and phase shift of a structured light field:
the light is illuminated by a laser or a Light Emitting Diode (LED) light source, and a structured light illumination light field with light intensity meeting one-dimensional cosine function distribution is formed on a sample through the modulation of a Spatial Light Modulator (SLM) or a Digital Micromirror Device (DMD) light field modulation device; the intensity of the structured light illumination field satisfies the distribution of formula (1):
where r denotes the two-dimensional object coordinate of the sample plane, I0Representing the mean value of the intensity of the structured light field, m representing the degree of modulation, k0The spatial frequency is represented by a representation of,representing an initial phase;
controlling the light field modulation device to generate 2 structured light fields with different spatial directions and an included angle theta of 90 degrees between the adjacent directions in a sample plane; in each spatial direction by the amount of phase shiftGenerating 3 structured light fields with different phase shifts, the phases of which are respectivelyAndgenerating structural light fields at 6 different positions in sequence for illuminating and exciting the sample to generate a fluorescence signal;
step 2), collecting a fluorescence image by an area array digital camera:
the method comprises the following steps that (1) an illumination light field with 3 phase shift structures corresponding to 2 spatial directions is imaged by a fluorescence microscope system, 6 fluorescence images illuminated by structural light are sequentially recorded on an image surface of the imaging system by an area array digital camera CMOS or CCD, the fluorescence images are divided into 2 groups according to the spatial directions, each group comprises 3 different phases and are respectively recorded as: { D11(r’),D12(r’),D13(r’)}、{D21(r’),D22(r’),D23(r’)};
Step 3), super-resolution image reconstruction processing:
step 3.1), according to the light intensity average value I of the structured light field0Degree of modulation m, spatial frequency k02 initial phase in spatial directionAnd phase shift amount pi/2, generating a demodulation coefficient matrix shown in formula (5), and correspondingly dividing the demodulation coefficient matrix into 2 groups according to the spatial direction, wherein each group contains 3 different phases which are respectively recorded as: { c11(r’),c12(r’),c13(r’)}、{c21(r’),c22(r’),c23(r’)};
Step 3.2), calculating the 6 structural light illuminated fluorescence images collected in the step 2) and the demodulation coefficient matrix generated in the step 3.1) according to the sequence by a formula (6) to obtain an initial super-resolution image RSDR(r’):
Step 3.3), generating a new super-resolution equivalent point spread function P (r ') according to a formula (4) by using an optical system point spread function H (r') obtained theoretically or experimentally;
P(r′)=H(r′)[1+cos(2πk0r′)](4)
step 3.4), using the initial super-resolution image R obtained in step 3.2)SDR(R ') and the super-resolution equivalent point spread function P (R') generated in the step 3.3), and the deconvolution operation is completed to obtain the final super-resolution image RSIM(r’)。
5. A fast system for directly reconstructing a structured light illumination super-resolution image in a space domain comprises a processor and a memory, and is characterized in that: the memory has stored therein a computer program which, when run on the processor, performs the method of step 3) of claim 4.
6. A computer-readable storage medium characterized by: a computer program is stored which, when executed, implements the method of step 3) of claim 4.
7. A rapid method for directly reconstructing a structured light illumination super-resolution image in a space domain is characterized by comprising the following steps under the condition of nonlinear excitation response:
step 1), excitation and phase shift of a nonlinear structure light field:
the light is illuminated by a laser or a Light Emitting Diode (LED) light source, and a structured light illumination light field with light intensity meeting one-dimensional cosine function distribution is formed on a sample through the modulation of a Spatial Light Modulator (SLM) or a Digital Micromirror Device (DMD) light field modulation device; the intensity of the structured light illumination field satisfies the distribution of formula (1);
where r denotes the two-dimensional object coordinate of the sample plane, I0Representing the mean value of the intensity of the structured light field, m representing the degree of modulation, k0The spatial frequency is represented by a representation of,representing an initial phase;
by utilizing the nonlinear response characteristic of the fluorescent molecules, the intensity of the generated fluorescent signal light has the optical field distribution described by the formula (7):
where r denotes the two-dimensional object coordinate of the sample plane, I0Representing the mean value of the light intensity, M the harmonic order of the structured light field, bnIndicating the degree of modulation, k, of the nth harmonic0Which represents the spatial frequency of the fundamental frequency,representing an initial phase;
controlling the optical field modulation device to generate (2M +1) structural optical fields with different spatial directions and an included angle theta between adjacent directions being 360 degrees/2M +1 on a sample plane; in each spatial direction, it is necessary to shift the phase by the amountGenerating (2M +1) structured light fields with different phase shifts; in the case of harmonic order M being 2, the light intensity distribution of the fluorescence signal generated by the nonlinear excitation response satisfies the structural light field described by formula (8):
in this case, it is necessary to generate 5 structured light fields with different spatial directions and an angle θ between adjacent directions of 72 ° on the sample plane; in each spatial direction, it is necessary to shift the phase by the amountGenerating 5 structured light fields with different phase shifts; so as to generate 25 excited fluorescence structure light fields at different positions;
2) acquiring a fluorescence image by an area array digital camera:
corresponding to 5 phase shift structure light fields in 5 spatial directions, and imaging by a fluorescence microscope systemOn the image surface of the imaging system, 25 fluorescence images illuminated by the structured light are sequentially obtained by recording by an area array digital camera CMOS or CCD, the fluorescence images are divided into 5 groups according to the spatial direction, each group contains 5 different phases which are respectively recorded as: { D11(r’),D12(r’),……,D15(r’)}、{D21(r’),D22(r’),……,D25(r’)}、……、{D51(r’),D52(r’),……,D55(r ') }, where r' represents image plane two-dimensional image coordinates;
step 3), super-resolution image reconstruction processing:
step 3.1), according to the light intensity average value I of the structured light field0Modulation b of each harmonic wave, M20、b1、b2Fundamental frequency k of space0Initial phase in 5 spatial directionsAnd phase shift amount 2 pi/5, obtaining a demodulation coefficient matrix described by formula (9), and correspondingly dividing the demodulation coefficient matrix into 5 groups according to the spatial direction, wherein each group contains 5 different phases which are respectively recorded as: { c11(r’),c12(r’),……,c15(r’)}、{c21(r’),c22(r’),……,c25(r’)}、……、{c51(r’),c52(r’),……,c55(r’)}:
Step 3.2), calculating the 25 structured light illuminated fluorescence images acquired in the step 2) and the demodulation coefficient matrix generated in the step 3.1) according to the sequence through a formula (10) to obtain an initial super-resolution image RSDR(r’):
Step 3.3), generating a new super-resolution equivalent point spread function P (r ') according to a formula (11) by using an optical system point spread function H (r') obtained theoretically or experimentally:
P(r′)=H(r′)×{b0+b1cos[2πk0r′]+b2cos[2(2πk0r′)]} (11)
step 3.4), using the initial super-resolution image R obtained in step 3.2)SDR(R ') and the super-resolution equivalent point spread function P (R') generated in the step 3.3), and the deconvolution operation is completed to obtain the final super-resolution image RSIM(r’)。
8. A fast system for directly reconstructing a structured light illumination super-resolution image in a space domain comprises a processor and a memory, and is characterized in that: the memory has stored therein a computer program which, when run on the processor, performs the method of step 3) of claim 7.
9. A computer-readable storage medium characterized by: a computer program is stored which, when executed, implements the method of step 3) of claim 7.
10. A rapid method for directly reconstructing a structured light illumination super-resolution image in a space domain is characterized by comprising the following steps under the condition of nonlinear excitation response:
step 1), excitation and phase shift of a nonlinear structure light field:
the light is illuminated by a laser or a Light Emitting Diode (LED) light source, and a structured light illumination light field with light intensity meeting one-dimensional cosine function distribution is formed on a sample through the modulation of a Spatial Light Modulator (SLM) or a Digital Micromirror Device (DMD) light field modulation device; the intensity of the structured light illumination field satisfies the distribution of formula (1);
where r denotes the two-dimensional object coordinate of the sample plane, I0Representing the mean value of the intensity of the structured light field, m representing the degree of modulation, k0The spatial frequency is represented by a representation of,representing an initial phase;
by utilizing the nonlinear response characteristic of the fluorescent molecules, the intensity of the generated fluorescent signal light has the optical field distribution described by the formula (7):
where r denotes the two-dimensional object coordinate of the sample plane, I0Representing the mean value of the light intensity, M the harmonic order of the structured light field, bnIndicating the degree of modulation, k, of the nth harmonic0Which represents the spatial frequency of the fundamental frequency,representing an initial phase;
in the case of harmonic order M being 3, the light intensity distribution of the fluorescence signal generated by the nonlinear excitation response satisfies the structural light field described by formula (12):
controlling the light field modulation device to generate 7 structured light fields with different spatial directions and an included angle theta of 51.4 degrees between the adjacent directions on a sample plane; in each spatial direction, it is necessary to shift the phase by the amountGenerating 7 structured light fields with different phase shifts; so as to generate 49 light fields of the excited fluorescence structure at different positions;
step 2), collecting a fluorescence microscopic image by an area array digital camera:
corresponding to 7 phase shift structure light fields in 7 spatial directions, after imaging by a fluorescence microscope system, sequentially recording 49 fluorescence images illuminated by structural light on an image surface of the imaging system by a CMOS or CCD (complementary metal oxide semiconductor) of an area array digital camera, and dividing the fluorescence images into 7 groups according to the spatial directions, wherein each group comprises 7 different structural light fieldsThe phases, respectively noted: { D11(r’),D12(r’),……,D17(r’)}、{D21(r’),D22(r’),……,D27(r’)}、……、{D71(r’),D72(r’),……,D77(r ') }, where r' represents image plane two-dimensional image coordinates;
step 3), super-resolution image reconstruction processing:
step 3.1), according to the light intensity average value I of the structured light field0Modulation b of each harmonic wave 30、b1、b2、b3Fundamental frequency k of space0Initial phase in 7 spatial directions And phase shift amount 2 pi/7, obtaining a demodulation coefficient matrix described by formula (13), and correspondingly dividing the demodulation coefficient matrix into 7 groups according to the space direction, wherein each group contains 7 different phases which are respectively recorded as: { c11(r’),c12(r’),……,c17(r’)}、{c21(r’),c22(r’),……,c27(r’)}、……、{c71(r’),c72(r’),……,c77(r’)}:
Step 3.2), calculating the 49 structural light illuminated fluorescence images acquired in the step 2) and the demodulation coefficient matrix generated in the step 3.1) according to the sequence by a formula (14) to obtain an initial super-resolution image RSDR(r’):
Step 3.3), generating a new super-resolution equivalent point spread function P (r ') according to a formula (15) by using an optical system point spread function H (r') obtained theoretically or experimentally:
P(r′)=H(r′)×{b0+b1cos[2πk0r′]+b2cos[2(2πk0r′)]+b3cos[3(2πk0r′)]} (15)
step 3.4), using the initial super-resolution image R obtained in step 3.2)SDR(R ') and the super-resolution equivalent point spread function P (R') generated in the step 3.3), and the deconvolution operation is completed to obtain the final super-resolution image RSIM(r’)。
11. A fast system for directly reconstructing a structured light illumination super-resolution image in a space domain comprises a processor and a memory, and is characterized in that: the memory has stored therein a computer program which, when run on the processor, performs the method of step 3) of claim 10.
12. A computer-readable storage medium characterized by: a computer program is stored which, when executed, implements the method of step 3) of claim 10.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911238624.6A CN111077121B (en) | 2019-12-06 | 2019-12-06 | Rapid method and system for directly reconstructing structured light illumination super-resolution image in space domain |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911238624.6A CN111077121B (en) | 2019-12-06 | 2019-12-06 | Rapid method and system for directly reconstructing structured light illumination super-resolution image in space domain |
Publications (2)
Publication Number | Publication Date |
---|---|
CN111077121A true CN111077121A (en) | 2020-04-28 |
CN111077121B CN111077121B (en) | 2020-11-17 |
Family
ID=70313213
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201911238624.6A Active CN111077121B (en) | 2019-12-06 | 2019-12-06 | Rapid method and system for directly reconstructing structured light illumination super-resolution image in space domain |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN111077121B (en) |
Cited By (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111458318A (en) * | 2020-05-12 | 2020-07-28 | 西安交通大学 | Super-resolution imaging method and system utilizing square lattice structure light illumination |
CN111948296A (en) * | 2020-07-20 | 2020-11-17 | 南昌大学 | Method for improving axial resolution of photoacoustic microscopic imaging system based on axial modulation |
CN113112405A (en) * | 2021-04-12 | 2021-07-13 | 广州超视计生物科技有限公司 | Self-adaptive correction method of super-resolution microscope image and SIM-ODT (subscriber identity module-ODT) bimodal system |
CN113466192A (en) * | 2021-06-16 | 2021-10-01 | 浙大宁波理工学院 | Ultra-high-speed real-time super-resolution microscopic imaging method |
WO2021227292A1 (en) * | 2020-05-12 | 2021-11-18 | 北京大学 | Direct structured illumination microscopy reconstruction method |
CN113670878A (en) * | 2021-08-25 | 2021-11-19 | 西安交通大学 | Super-resolution structured light illumination microscopic imaging method based on space-frequency domain hybrid reconstruction |
CN113947565A (en) * | 2021-09-03 | 2022-01-18 | 中国科学院西安光学精密机械研究所 | Structured light illumination super-resolution imaging gene detection method based on deep learning |
CN115839936A (en) * | 2022-12-12 | 2023-03-24 | 之江实验室 | Phase-locked detection-based structured light illumination super-resolution microscopic imaging reconstruction method |
Citations (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101655601A (en) * | 2008-08-22 | 2010-02-24 | 麦克奥迪实业集团有限公司 | Light microscope imaging method and system based on DMD structure |
CN102540446A (en) * | 2011-12-28 | 2012-07-04 | 中国科学院西安光学精密机械研究所 | High-speed structure illumination optical microscope system and method based on digital micromirror device |
CN103645136A (en) * | 2013-11-22 | 2014-03-19 | 深圳先进技术研究院 | Method and device for improving multiphoton fluorescence microscope imaging resolution |
CN104515759A (en) * | 2014-12-16 | 2015-04-15 | 中国科学院苏州生物医学工程技术研究所 | Non-linear structure light illumination microscopic imaging method and system |
CN106054380A (en) * | 2016-07-15 | 2016-10-26 | 中国科学院苏州生物医学工程技术研究所 | Light section fluorescent image lighting method through FPGA acceleration processing structured light |
JP2016537624A (en) * | 2013-10-21 | 2016-12-01 | ユニバーシティ オブ レスター | Improvement of or related to super-resolution microscope |
CN106770147A (en) * | 2017-03-15 | 2017-05-31 | 北京大学 | A kind of Structured Illumination super-resolution micro imaging system and its imaging method |
CN107180411A (en) * | 2017-05-19 | 2017-09-19 | 中国科学院苏州生物医学工程技术研究所 | A kind of image reconstructing method and system |
CN107389631A (en) * | 2017-04-28 | 2017-11-24 | 中国科学院生物物理研究所 | The multi-modal Structured Illumination super-resolution micro imaging system of high speed polychrome and its method |
CN107966826A (en) * | 2017-12-27 | 2018-04-27 | 中国科学院半导体研究所 | A kind of small-scale structure optical illumination super-resolution micro imaging system |
WO2019163243A1 (en) * | 2018-02-20 | 2019-08-29 | 浜松ホトニクス株式会社 | Image processing method, image processing device, and image processing program |
-
2019
- 2019-12-06 CN CN201911238624.6A patent/CN111077121B/en active Active
Patent Citations (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101655601A (en) * | 2008-08-22 | 2010-02-24 | 麦克奥迪实业集团有限公司 | Light microscope imaging method and system based on DMD structure |
CN102540446A (en) * | 2011-12-28 | 2012-07-04 | 中国科学院西安光学精密机械研究所 | High-speed structure illumination optical microscope system and method based on digital micromirror device |
JP2016537624A (en) * | 2013-10-21 | 2016-12-01 | ユニバーシティ オブ レスター | Improvement of or related to super-resolution microscope |
CN103645136A (en) * | 2013-11-22 | 2014-03-19 | 深圳先进技术研究院 | Method and device for improving multiphoton fluorescence microscope imaging resolution |
CN104515759A (en) * | 2014-12-16 | 2015-04-15 | 中国科学院苏州生物医学工程技术研究所 | Non-linear structure light illumination microscopic imaging method and system |
CN106054380A (en) * | 2016-07-15 | 2016-10-26 | 中国科学院苏州生物医学工程技术研究所 | Light section fluorescent image lighting method through FPGA acceleration processing structured light |
CN106770147A (en) * | 2017-03-15 | 2017-05-31 | 北京大学 | A kind of Structured Illumination super-resolution micro imaging system and its imaging method |
CN107389631A (en) * | 2017-04-28 | 2017-11-24 | 中国科学院生物物理研究所 | The multi-modal Structured Illumination super-resolution micro imaging system of high speed polychrome and its method |
CN107180411A (en) * | 2017-05-19 | 2017-09-19 | 中国科学院苏州生物医学工程技术研究所 | A kind of image reconstructing method and system |
CN107966826A (en) * | 2017-12-27 | 2018-04-27 | 中国科学院半导体研究所 | A kind of small-scale structure optical illumination super-resolution micro imaging system |
WO2019163243A1 (en) * | 2018-02-20 | 2019-08-29 | 浜松ホトニクス株式会社 | Image processing method, image processing device, and image processing program |
Non-Patent Citations (3)
Title |
---|
XING ZHOU 等: ""Image recombination transform algorithm for superresolution structured illumination microscopy"", 《JOURNAL OF BIOMEDICAL OPTICS》 * |
周兴 等: ""结构光照明显微中的超分辨图像重建研究"", 《光学学报》 * |
赵天宇 等: ""结构光照明显微中的偏振控制"", 《物理学报》 * |
Cited By (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111458318B (en) * | 2020-05-12 | 2021-06-22 | 西安交通大学 | Super-resolution imaging method and system utilizing square lattice structure light illumination |
WO2021227292A1 (en) * | 2020-05-12 | 2021-11-18 | 北京大学 | Direct structured illumination microscopy reconstruction method |
CN111458318A (en) * | 2020-05-12 | 2020-07-28 | 西安交通大学 | Super-resolution imaging method and system utilizing square lattice structure light illumination |
CN111948296A (en) * | 2020-07-20 | 2020-11-17 | 南昌大学 | Method for improving axial resolution of photoacoustic microscopic imaging system based on axial modulation |
CN113112405B (en) * | 2021-04-12 | 2022-04-12 | 广州超视计生物科技有限公司 | Self-adaptive correction method of super-resolution microscope image and SIM-ODT (subscriber identity module-ODT) bimodal system |
CN113112405A (en) * | 2021-04-12 | 2021-07-13 | 广州超视计生物科技有限公司 | Self-adaptive correction method of super-resolution microscope image and SIM-ODT (subscriber identity module-ODT) bimodal system |
CN113466192A (en) * | 2021-06-16 | 2021-10-01 | 浙大宁波理工学院 | Ultra-high-speed real-time super-resolution microscopic imaging method |
CN113670878A (en) * | 2021-08-25 | 2021-11-19 | 西安交通大学 | Super-resolution structured light illumination microscopic imaging method based on space-frequency domain hybrid reconstruction |
CN113670878B (en) * | 2021-08-25 | 2022-09-27 | 西安交通大学 | Super-resolution structured light illumination microscopic imaging method based on space-frequency domain hybrid reconstruction |
CN113947565A (en) * | 2021-09-03 | 2022-01-18 | 中国科学院西安光学精密机械研究所 | Structured light illumination super-resolution imaging gene detection method based on deep learning |
CN113947565B (en) * | 2021-09-03 | 2023-04-18 | 中国科学院西安光学精密机械研究所 | Structured light illumination super-resolution imaging gene detection method based on deep learning |
CN115839936A (en) * | 2022-12-12 | 2023-03-24 | 之江实验室 | Phase-locked detection-based structured light illumination super-resolution microscopic imaging reconstruction method |
CN115839936B (en) * | 2022-12-12 | 2023-07-18 | 之江实验室 | Structured light illumination super-resolution microscopic imaging reconstruction method based on phase-locked detection |
Also Published As
Publication number | Publication date |
---|---|
CN111077121B (en) | 2020-11-17 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN111077121B (en) | Rapid method and system for directly reconstructing structured light illumination super-resolution image in space domain | |
CN107389631B (en) | High-speed multicolor multi-modal structured light illumination super-resolution microscopic imaging system and method thereof | |
US8019136B2 (en) | Optical sectioning microscopy | |
Kumar et al. | Common-path multimodal three-dimensional fluorescence and phase imaging system | |
Dan et al. | Structured illumination microscopy for super-resolution and optical sectioning | |
CN113670878B (en) | Super-resolution structured light illumination microscopic imaging method based on space-frequency domain hybrid reconstruction | |
CN110954521B (en) | Wide-field super-resolution microscopic imaging method and system thereof | |
CN105929560A (en) | Broadband far field super-resolution imaging apparatus | |
CN109712072B (en) | Fringe illumination Fourier domain iterative updating super-resolution microscopic imaging method based on total internal reflection | |
CN111458318B (en) | Super-resolution imaging method and system utilizing square lattice structure light illumination | |
Chen et al. | Superresolution structured illumination microscopy reconstruction algorithms: a review | |
US20200150043A1 (en) | Accelerated methods and apparatuses for three-dimensional microscopy with structured illumination | |
JP5085608B2 (en) | Wide-field super-resolution optical microscope using a spatial light modulator | |
CN110954523B (en) | Two-photon scanning structure light microscopic imaging method and device | |
US20220076379A1 (en) | Method for super-resolution evaluation of microscope images illuminated in a structured manner and microscope having structured illumination | |
Dan et al. | Rapid image reconstruction of structured illumination microscopy directly in the spatial domain | |
CN114113019A (en) | Array scanning super-resolution microscopic imaging device, method and equipment based on multiple signal classification algorithm and storage medium | |
CN114594588B (en) | Structured light illumination microscopic device and method based on grating projection and SLM phase shift | |
Stemmer et al. | Widefield fluorescence microscopy with extended resolution | |
CN114967092B (en) | Super-high-speed structured light illumination super-resolution microscopic imaging device based on compressed sensing | |
CN111024664B (en) | Imaging method of structured light illumination super-resolution microscopic chip | |
US11947098B2 (en) | Multi-focal light-sheet structured illumination fluorescence microscopy system | |
JP2017523439A (en) | Double structured illumination with modulated phase and intensity | |
JP7390732B2 (en) | Programmable multipoint illuminators, confocal filters, confocal microscopes, and how to operate confocal microscopes | |
CN113568294B (en) | Holographic optical tweezers fusion structure light illumination microscopic system and method |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant | ||
TR01 | Transfer of patent right | ||
TR01 | Transfer of patent right |
Effective date of registration: 20210517 Address after: 518083 the comprehensive building of Beishan industrial zone and 11 2 buildings in Yantian District, Shenzhen, Guangdong. Patentee after: Shenzhen Huada Zhizao Technology Co., Ltd Address before: 710119, No. 17, information Avenue, new industrial park, hi tech Zone, Shaanxi, Xi'an Patentee before: Xi'an Institute of Optics and precision machinery, Chinese Academy of Sciences |