CN113112405B - Self-adaptive correction method of super-resolution microscope image and SIM-ODT (subscriber identity module-ODT) bimodal system - Google Patents

Self-adaptive correction method of super-resolution microscope image and SIM-ODT (subscriber identity module-ODT) bimodal system Download PDF

Info

Publication number
CN113112405B
CN113112405B CN202110386643.4A CN202110386643A CN113112405B CN 113112405 B CN113112405 B CN 113112405B CN 202110386643 A CN202110386643 A CN 202110386643A CN 113112405 B CN113112405 B CN 113112405B
Authority
CN
China
Prior art keywords
sample
image
sim
super
original
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110386643.4A
Other languages
Chinese (zh)
Other versions
CN113112405A (en
Inventor
莫燕权
陈良怡
毛珩
范骏超
丰帆
梁林涛
屠锐
杜珂
杨宏润
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangzhou Chaoshiji Biotechnology Co ltd
Original Assignee
Guangzhou Chaoshiji Biotechnology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangzhou Chaoshiji Biotechnology Co ltd filed Critical Guangzhou Chaoshiji Biotechnology Co ltd
Priority to CN202110386643.4A priority Critical patent/CN113112405B/en
Publication of CN113112405A publication Critical patent/CN113112405A/en
Application granted granted Critical
Publication of CN113112405B publication Critical patent/CN113112405B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/58—Optics for apodization or superresolution; Optical synthetic aperture systems

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Optics & Photonics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Microscoopes, Condenser (AREA)
  • Investigating, Analyzing Materials By Fluorescence Or Luminescence (AREA)

Abstract

The invention discloses a self-adaptive correction method of a super-resolution microscope image and an SIM-ODT bimodal system, wherein the method comprises the following steps: step 1a, reconstructing to obtain an SIM super-resolution image with artifacts; step 2a, dividing the SIM super-resolution image and the original fluorescence image into a plurality of first SIM super-resolution sub-images and original fluorescence sub-images; step 3a, finding out an imaging focal plane corresponding to each original fluorescent sub-image containing a fluorescent signal and a sample layer number corresponding to the imaging focal plane through three-dimensional refractive index distribution, and arranging point light sources at the center of each original fluorescent sub-image containing the fluorescent signal and an adjacent region of the center; step 4a, calculating the light field distribution of the point light source transmitted to the surface of the sample from the imaging focal plane corresponding to the original fluorescent sub-image; step 5a, updating an optical transfer function; and 6a, reconstructing to obtain a second SIM super-resolution sub-image of the original fluorescence sub-image. The invention can carry out self-adaptive correction on all aberrations and artificial artifacts of the super-resolution microscope reconstructed image.

Description

Self-adaptive correction method of super-resolution microscope image and SIM-ODT (subscriber identity module-ODT) bimodal system
Technical Field
The invention relates to the technical field of super-resolution microscopic imaging, in particular to a self-adaptive correction method of a super-resolution microscope image and an SIM-ODT (subscriber identity module-ODT) bimodal system.
Background
The SIM (structured light microscope) breaks through the optical diffraction limit by adopting a sinusoidal illumination mode with multiple directions and multiple phases for spectral modulation, so as to achieve the purpose of super-resolution (super-resolution) imaging. Meanwhile, compared with other super-resolution technologies such as an stimulated radiation depletion microscope STED and a single-molecule positioning microscope PALM/STROM, the SIM has the advantages of small sample specificity requirement, small photon number, small phototoxicity, high imaging speed and the like, and becomes one of the most widely applied super-resolution technologies in living cell research. However, despite its many advantages, SIM is susceptible to reconstruction artifacts: inaccurate parameter estimation, low image signal-to-noise ratio, large optical transfer function error, non-negligible background fluorescence of an out-of-focus plane and the like all bring difficulty to reconstruction of the SIM super-resolution image. For a sample with a large thickness, the refractive index distribution inside the sample is very different because the distribution of organelles is not uniform. These structural differences cause scattering of the SIM illumination laser and the excited fluorescence, which in turn causes imaging aberrations to the microscope, which is also the most important factor limiting the imaging depth of the SIM, so that the current SIM is still limited to imaging in the range of thin sample or cell shallow region.
If the problem of artifacts of deep super-resolution reconstructed images of SIM thick samples can be solved, the application range of the SIM can be greatly expanded, and the experimental research progress of cell biology is facilitated. In a conventional SIM super-resolution image reconstruction algorithm, an Optical Transfer Function (OTF) used for reconstruction is basically derived from an OTF constructed by an ideal physical model or an OTF obtained by shooting with fluorescent beads, and is an OTF which is too ideal and unmatched for an actual biological sample. In the conventional method for solving the super-resolution image artifact, there is a method for directly adding a regular term in an image reconstruction algorithm model to further constrain the process, and there is also a method for detecting aberration by adding a wavefront sensor in an optical path and then correcting the wavefront aberration by using a deformable mirror by using an adaptive optics means, or directly correcting the aberration in the optical path by using a method without wavefront sensing. However, the method of adding a regularization term to the reconstruction model requires that the original fluorescence image of the SIM modality acquired by the microscope and the parameters used for reconstruction, such as OTF, are indistinguishable, which is obviously difficult to achieve for thick samples. However, the method of adaptive optical correction can only correct global aberration, but cannot correct local aberration.
Disclosure of Invention
The invention aims to provide a self-adaptive correction method of a super-resolution microscope image and an SIM-ODT dual-mode system, which can carry out self-adaptive correction on various aberrations and artificial artifacts of a super-resolution microscope reconstructed image.
In order to achieve the above object, the present invention provides an adaptive correction method for a super-resolution microscope image, including:
the method of adaptively correcting a SIM super-resolution image from an imaging probe path using a three-dimensional refractive index distribution obtained through an original phase image of an ODT (optical diffraction tomography) mode of a sample includes:
step 1a, reconstructing to obtain an initial SIM super-resolution image with an artifact according to an optical transfer function and an original fluorescence image of the sample in an SIM mode;
step 2a, dividing the initial SIM super-resolution image into a plurality of first SIM super-resolution sub-images according to a first dividing mode, dividing the original fluorescent image into a plurality of original fluorescent sub-images according to the first dividing mode, and overlapping the edges of the adjacent sub-images;
step 3a, finding out an imaging focal plane corresponding to each original fluorescence sub-image containing a fluorescence signal and a sample layer number corresponding to the imaging focal plane along an optical axis z axis of the SIM mode through the three-dimensional refractive index distribution, and arranging a point light source at the center of each original fluorescence sub-image containing the fluorescence signal and a selected point of an adjacent region of the center;
step 4a, calculating a first light field distribution U of the point light source transmitted to the surface of the sample from the imaging focal plane corresponding to the original fluorescent subimage according to the three-dimensional refractive index distributionL1(ρ), ρ ═ (x, y) is the two-dimensional spatial coordinate perpendicular to z, and L is the total number of layers of the sample from the imaging focal plane corresponding to the original fluorescence sub-image to the outer surface of the sample;
step 5a, according to UL1(p), calculating a first coherent transfer function of the imaging focal plane corresponding to the original fluorescence sub-image
Figure GDA0003515564190000021
And is composed of
Figure GDA0003515564190000022
Updating the first optical transfer function H1(ξ), ξ being a two-dimensional coordinate in the frequency domain;
step 6a, according to H1(ξ) and the corresponding original fluorescence sub-image, and reconstructing again to obtain a second SIM super-resolution sub-image of the original fluorescence sub-image;
and 7a, splicing the second SIM super-resolution sub-image reconstructed in the step 6a and the first SIM super-resolution sub-image corresponding to the region where the original fluorescence sub-image without the fluorescence signal is located into a final complete SIM super-resolution image.
Further, in said step 5a
Figure GDA0003515564190000031
Obtained by one of the following two methods:
method one, according to the focal length of the second objective lens, the U is adjustedL1(rho) is positively propagated to the pupil surface position by utilizing a Fresnel diffraction formula to obtain
Figure GDA0003515564190000032
The second lens is used for receiving ODT modal laser after passing through the sample, irradiating the sample after the laser generated by the SIM super-resolution laser lighting module passes through the sample, and generating original fluorescence in an SIM mode generated by the sample; wherein the second lens is used for receiving ODT mode laser after passing through the sample, original fluorescence of SIM mode generated by the sample and allowing the SIM mode laser to pass through and irradiate the sample;
secondly, calculating U by utilizing refocusing back propagation provided by the formula (1)L1(ρ) obtaining
Figure GDA0003515564190000033
Figure GDA0003515564190000034
Figure GDA0003515564190000035
In the formula of UPSF(p) is UL1(ρ) light field distribution propagating back to the imaging focal plane in the opposite direction, i.e. the coherence point spread function, ρLIs the coordinate of any place on the outer surface of the sample, zLCharacterizing the pupil function, U, for the physical distance of the imaging focal plane to the outer surface of the sample, P (-), P (-. cndot.)L(ρL) Is the light field distribution of the outer surface of the sample, j is the unit of imaginary number, kmFt {. is a Fourier transform of a wave vector of a light field of the illumination laser of the SIM mode in an environment.
Further, U in the step 4aL1The (ρ) acquisition method specifically includes:
step 41a, setting the point light source as an ideal spherical wave;
step 42a, calculating a scattering potential f (r) by using a formula (3) according to the three-dimensional refractive index distribution n (r);
Figure GDA0003515564190000036
Figure GDA0003515564190000037
in the formula, nmIs the average refractive index, k, of the environment in which the sample is locatedmWave vector of light field of illumination laser in environment in SIM mode, lambda0A wavelength of an illumination laser in the SIM mode;
step 43a, according to f (r), calculating UL1(ρ)。
Further, U in the step 43aL1(ρ) is solved by using a Lippmann-schwigger model in an integral form, or using a multilayer beam propagation model provided by the following equations (5) to (7):
Ul(ρ，(l+1)Δz)＝Ft-1{Pprop(ξ，Δz)·Ft{Ul(ρ，lΔz)}} (5)
Figure GDA0003515564190000041
Figure GDA0003515564190000042
Figure GDA0003515564190000043
Figure GDA0003515564190000044
in the formula of Ul(ρ, (l +1) Δ z) is the point source from the original fluorescenceThe corresponding imaging focal plane of the optical sub-image directly propagates to the light field distribution, U, of the l-th layer of the sample without sample effectl(p, l Δ z) is the total light field distribution of the point light source propagating from the corresponding imaging focal plane of the original fluorescence sub-image to layer l-1 of the sample,
Figure GDA0003515564190000045
is Ul(ρ, (l) Δ z) the scattered light field, U, generated by the interaction with the sample after passing through the l-th layer of said samplel+1(ρ, (L +1) Δ z) is the total light field distribution of the point light source propagating from the imaging focal plane corresponding to the original fluorescence sub-image to the L-th layer of the sample, L is the number of layers of the point light source from the imaging focal plane corresponding to the original fluorescence sub-image to the outer surface of the sample, L is 1 … …, L, z is the coordinate distribution of the optical axis of the SIM-mode, Δ z is the thickness of each layer of the sample, Ft {. cndot.) represents the fourier transform, Ft {. cndot-1{. represents an inverse Fourier transform, Pprop(xi, Δ z) is an angular spectrum propagation function represented by formula (8), G (xi, Δ z) is a green function represented by formula (9), fl(p) is the scattering potential of the point light source propagating from the imaging focal plane corresponding to the original fluorescent sub-image to the ith layer of the sample, when l is 1, U is1(rho, Delta z) is taken as USPH(ρ，Δz)。
The invention also provides a self-adaptive correction method of the super-resolution microscope image, which comprises the following steps:
the method for adaptively correcting a SIM super-resolution image from an illumination path using a three-dimensional refractive index distribution obtained through an original phase image of an ODT modality of a sample includes:
step 1b, estimating and solving illumination parameters of each illumination mode according to the original fluorescence image of the SIM mode of the sample;
step 2b, generating uniform ideal sinusoidal illumination mode distribution at the outer surface of the sample according to the prior information of the illumination mode in sinusoidal distribution and the illumination parameters obtained by the first module;
step 3b, according to the three-dimensional refractive index distributionCalculating the total number L of layers from the imaging focal plane corresponding to the original fluorescence image to the outer surface of the sample along the optical axis z axis of the SIM mode and the second light field distribution U of the ideal sine distribution light source transmitted from the outer surface of the sample to the imaging focal plane of the original fluorescence imageL2(ρ) where ρ is (x, y) a two-dimensional spatial coordinate perpendicular to the z-axis;
step 4b, according to UL2(ρ) obtaining the illumination pattern after scattering distortion;
and 5b, iterating the illumination mode after the scattering distortion by using the optimization model represented by the formula (10), and reconstructing to obtain the optimal SIM super-resolution image
Figure GDA0003515564190000051
Figure GDA0003515564190000052
Where num is the total number of frames of the original fluorescence image in each period of the illumination pattern, DiFor the i frame of the original fluorescence image, H is a second optical transfer function H2(xi) point spread function, g, obtained after inverse Fourier transformiComprises the following steps: gi(ρ)＝I′i(ρ)·o(ρ)，I′i(ρ) and I'iIs DiCorresponding to the illumination mode after scattering distortion, Hess (o) represents Hessian matrix, | · |. includes1The L1 norm of the expression matrix, o and o (rho) are SIM super-resolution images to be reconstructed, lambda is a data fidelity term coefficient, alpha is a Hessian continuity constraint coefficient, beta is a sparse term constraint coefficient, and xi is a two-dimensional coordinate in a frequency domain.
Further, the illumination mode after the scattering distortion in the step 4b is represented as I' (ρ), which is obtained by equation (12):
I′(ρ)＝|UL2(ρ)|2 (12)。
further, U in the step 3bL2The (ρ) acquisition method specifically includes:
step 31b, calculating a scattering potential f (r) by using the formula (3) according to the three-dimensional refractive index distribution n (r);
Figure GDA0003515564190000053
Figure GDA0003515564190000054
in the formula, nmIs the average refractive index, k, of the environment in which the sample is locatedmWave vector of light field of illumination laser in environment in SIM mode, lambda0A wavelength of an illumination laser in the SIM mode;
step 32b, according to f (r), calculating UL2(ρ)。
Further, U in the step 32bL2(ρ) is solved by using a Lippmann-schwigger model in an integral form or using a multilayer beam propagation model provided by the following equations (5) to (7):
Ul(ρ，(l+1)Δz)＝Ft-1{Pprop(ξ，Δz)·Ft{Ul(ρ，lΔz)}} (5)
Figure GDA0003515564190000055
Figure GDA0003515564190000056
Figure GDA0003515564190000061
Figure GDA0003515564190000062
in the formula of Ul(ρ, (l +1) Δ z) is the direct propagation of the ideal sinusoidal illumination mode distribution from the outer surface of the sample to the sample without sample interactionOptical field distribution of the l-th layer, Ul(p, l Δ z) is the total light field distribution for the ideal sinusoidal illumination mode distribution to propagate from the outer surface of the sample to the l-1 st layer of the sample,
Figure GDA0003515564190000063
is Ul(ρ, (l) Δ z) the scattered light field, U, generated by the interaction with the sample after passing through the l-th layer of said samplel+1(ρ, (L +1) Δ z) is the total light field distribution of the ideal sinusoidal illumination mode distribution propagating from the outer surface of the sample to the L-th layer of the sample, L is the number of layers of the imaging focal plane corresponding to the ideal sinusoidal illumination mode distribution from the original fluorescence image of the SIM modality of the sample, L-1 … …, L, z is the coordinate distribution of the optical axis of the SIM modality, Δ z is the thickness of each layer of the sample, Ft {. cndot } represents the fourier transform, Ft {. cndot-1{. represents an inverse Fourier transform, Pprop(xi, Δ z) is an angular spectrum propagation function represented by formula (8), G (xi, Δ z) is a green function represented by formula (9), fl(p) is the scattering potential of the ideal sinusoidal illumination mode distribution propagating from the imaging focal plane corresponding to the original fluorescence image of the SIM modality of the sample to the ith layer of the sample, when l is 1, U1(ρ, Δ z) takes the ideal sinusoidal illumination pattern generated in step 2 b.
Further, H in said step 5b2(xi) is obtained by one of two ways:
the method comprises the steps of firstly, constructing an optical transfer function by using an ideal model to obtain;
and secondly, acquiring a point spread function H (rho) by fluorescent bead shooting, and acquiring H by using the H (rho)2(ξ)，H2(ξ) ═ Ft { h (ρ) }, Ft is the fourier transform.
Further, H in said step 5b2(ξ) is obtained by:
step 51b, using a three-dimensional refractive index profile obtained by an original phase image of an ODT mode of a sample;
step 52b, reconstructing to obtain an initial SIM super-resolution image with an artifact according to the optical transfer function and the original fluorescence image of the sample in the SIM mode;
step 53b, dividing the original fluorescence image into a plurality of original fluorescence sub-images according to the first division mode, wherein the edges of the adjacent sub-images are overlapped with each other;
step 54b, finding out an imaging focal plane corresponding to each original fluorescence sub-image containing the fluorescence signal and the number of sample layers corresponding to the imaging focal plane along the optical axis z axis of the SIM mode through the three-dimensional refractive index distribution, and arranging a point light source at the center of each original fluorescence sub-image containing the fluorescence signal and a selected point of an adjacent region of the center;
step 55b, calculating a first light field distribution U of the point light source transmitted to the surface of the sample from the imaging focal plane corresponding to the original fluorescent sub-image according to the three-dimensional refractive index distributionL1(ρ), ρ ═ (x, y) is the two-dimensional spatial coordinate perpendicular to z, and L is the total number of layers of the sample from the imaging focal plane corresponding to the original fluorescence sub-image to the outer surface of the sample;
step 56b, according to UL1(p), calculating a first coherent transfer function of the imaging focal plane corresponding to the original fluorescence sub-image
Figure GDA0003515564190000071
And is composed of
Figure GDA0003515564190000072
Updating the second optical transfer function H2(ξ)，
Figure GDA0003515564190000073
Figure GDA0003515564190000074
ξ is a two-dimensional coordinate in the frequency domain, which represents an autocorrelation calculation.
The invention also provides a SIM-ODT bimodal system, which comprises: an ODT lighting module for generating ODT mode laser light; the SIM super-resolution laser lighting module is used for generating SIM modal laser in sinusoidal distribution; the first objective lens is arranged between one side of the sample and the light outlet of the ODT lighting module and is used for irradiating the sample by the laser generated by the ODT lighting module; the second objective is arranged on the other side of the sample and is used for receiving ODT (optical time difference) mode laser after passing through the sample, passing the SIM mode laser and irradiating the sample, and passing original fluorescence of a SIM mode generated by the sample; the SIM super-resolution image acquisition module is used for acquiring an original fluorescence image of the SIM modality; the ODT image acquisition module is used for acquiring an original phase image of an ODT modality; the sample three-dimensional refractive index distribution calculation module is used for processing the original phase image by utilizing a reconstruction algorithm of diffraction tomography, and reconstructing to obtain the three-dimensional refractive index distribution of the unmarked sample; an adaptive image correction module for adaptively correcting the optical transfer function and/or distorted illumination pattern of the SIM modality using the adaptive correction method of the super resolution microscope image as described above; the 2D/3D-SIM super-resolution image reconstruction module is used for reconstructing the original fluorescence image and the optical transfer function according to the optical transfer function and/or the distorted illumination mode corrected by the self-adaptive image correction module to obtain a self-adaptive corrected SIM super-resolution image; and the bimodal image fusion module is used for fusing the SIM super-resolution image subjected to self-adaptive correction with the three-dimensional refractive index distribution according to a spatial position relationship, and simultaneously carrying out three-dimensional high-resolution dynamic imaging and deep specific two-dimensional or three-dimensional super-resolution dynamic imaging on the sample.
The invention obtains the three-dimensional refractive index information of the sample by utilizing the optical diffraction tomography mode in the dual-mode microscope system, and then adaptively corrects the two-dimensional or three-dimensional structure optical super-resolution image of the corresponding region from the illumination path and the detection path by utilizing the local refractive index information.
Drawings
FIG. 1 is a schematic diagram of a SIM-ODT bimodal system according to an embodiment of the present invention.
Fig. 2 is a block flow diagram of an adaptive calibration detection path according to an embodiment of the present invention.
Fig. 3 is a block flow diagram of an adaptive illumination path correction method according to an embodiment of the present invention.
Fig. 4 is a block diagram of a process for adaptively correcting a detection path and an illumination path according to an embodiment of the present invention.
Detailed Description
The invention is described in detail below with reference to the figures and examples.
As shown in fig. 1, an SIM-ODT bimodal system provided in an embodiment of the present invention includes an ODT illumination module, an SIM super-resolution laser illumination module, a first objective lens, a second objective lens, an SIM super-resolution image acquisition module, an ODT image acquisition module, a sample three-dimensional refractive index distribution calculation module, an adaptive image correction module, a 2D/3D-SIM super-resolution image reconstruction module, and a bimodal image fusion module, where:
the ODT lighting module is used for generating ODT modal laser and carrying out multi-angle unmarked diffraction chromatography lighting on the sample, and the used laser is required to have high coherence. Wherein the sample is typically a single cell thick, in the range of 10 μm to 20 μm.
The SIM super-resolution laser lighting module is used for generating SIM modal laser in sinusoidal distribution.
The first objective lens is arranged between one side of the sample and the light outlet of the ODT lighting module and is used for enabling the laser generated by the ODT lighting module to irradiate the sample.
The second objective lens is arranged on the other side of the sample and is used for receiving ODT mode laser after passing through the sample, allowing the SIM mode laser to pass through and irradiate the sample, and allowing original fluorescence of a SIM mode generated by the sample to pass through. In order to realize SIM super-resolution imaging, the second objective lens is a high-power objective lens, and the numerical aperture of the second objective lens is larger than or equal to that of the first objective lens, so that the laser of the ODT mode passing through the first objective lens can be received by the second objective lens.
And the SIM super-resolution image acquisition module is used for acquiring the original fluorescence image of the SIM modality.
The ODT image acquisition module is used for acquiring an original phase image of an ODT modality.
And the sample three-dimensional refractive index distribution calculation module is used for processing the original phase image by utilizing a reconstruction algorithm of diffraction tomography, such as Born approximate reconstruction, Rytov approximate reconstruction, multilayer propagation approximate reconstruction or other reconstruction methods, and reconstructing to obtain the unmarked three-dimensional refractive index distribution of the sample.
The adaptive image correction module is used for carrying out adaptive correction on the optical transfer function and/or the distorted illumination mode of the SIM modality by utilizing the imaging detection path and/or the illumination path. The imaging detection path refers to a process of emitting fluorescence from the sample, wherein the fluorescence propagates in the sample and is received by the second objective lens and then is acquired by the SIM super-resolution image acquisition module. The illumination path refers to a process that the SIM super-resolution laser illumination module emits laser, and the laser is irradiated to a sample imaging focal plane through the second objective lens to form a sine-distributed illumination mode.
And the 2D/3D-SIM super-resolution image reconstruction module carries out image reconstruction on the original fluorescence image and the optical transfer function according to the optical transfer function and/or the distorted illumination mode corrected by the self-adaptive image correction module to obtain a self-adaptively corrected SIM super-resolution image.
The image reconstruction method can be a traditional wiener deconvolution method, high-frequency components and low-frequency components in an image formed by the original fluorescence are decoupled, the high-frequency components are moved to positions corresponding to the illumination frequency, and finally the low-frequency components and the high-frequency components are integrated to obtain a super-resolution image; or the illumination mode is iterated integrally, and an optimal super-resolution image is found by utilizing an optimization method.
The bimodal image fusion module is used for fusing the SIM super-resolution image after self-adaptive correction and the three-dimensional refractive index distribution according to a spatial position relation, and simultaneously carrying out three-dimensional high-resolution dynamic imaging and deep specific two-dimensional or three-dimensional super-resolution dynamic imaging on the sample. Where "fusion" may be understood as interpolating the size of individual pixels of the ODT modality and the SIM modality to the same value, and then superimposing the results of the two modalities to show the effect of different channel colors in one image.
As shown in fig. 1 and fig. 2, an adaptive correction method for a super-resolution microscope image according to an embodiment of the present invention includes:
the method for adaptively correcting the SIM super-resolution image from the imaging detection path by using the three-dimensional refractive index distribution obtained through the original phase image of the ODT mode of the sample specifically comprises the following steps:
step 1a, reconstructing and obtaining an initial SIM super-resolution image with an artifact according to an optical transfer function and an original fluorescence image of the sample in the SIM mode. The optical transfer function can be constructed by using an ideal model or an optical transfer function obtained by shooting fluorescent beads. The reconstruction method can utilize a traditional wiener deconvolution image reconstruction method, namely: firstly, estimating illumination parameters such as illumination frequency, illumination phase and illumination modulation depth coefficient of each illumination direction by using spectral correlation; then, decoupling the high-frequency component and the low-frequency component in the original fluorescence image, and moving the high-frequency component to a position corresponding to the illumination frequency; and finally, integrating the low-frequency component and the high-frequency component to obtain an initial SIM super-resolution image with an artifact.
And 2a, dividing the initial SIM super-resolution image into a plurality of first SIM super-resolution sub-images according to a first dividing mode, dividing the original fluorescent image into a plurality of original fluorescent sub-images according to the first dividing mode, and overlapping the edges of the adjacent sub-images. The first division manner may be a uniform division manner, and may also be understood as: the divided image is divided into a plurality of pixels in the horizontal and vertical directions according to a grid division mode, wherein the number of the pixels in the horizontal and vertical directions is equal to or not less than 64 pixels. The number of pixels of the part where the edges of the adjacent sub-images overlap with each other is not less than 10% of the number of pixels of a single sub-image, so that the continuity of the subsequent image splicing can be ensured.
And 3a, finding out an imaging focal plane corresponding to each original fluorescence sub-image containing the fluorescence signal and the number of sample layers corresponding to the imaging focal plane along the optical axis z axis of the SIM mode through the three-dimensional refractive index distribution, and arranging a point light source at the center of each original fluorescence sub-image containing the fluorescence signal and a selected point of an adjacent region of the center. The selected point of the neighboring area in the center may be set as an eight-neighborhood diagonal point of the center point of the image, but is not limited thereto, and a person skilled in the art may select and set the selected point according to actual requirements. For convenience of calculation, the point light source can be replaced by an ideal spherical wave, as shown in formula (1):
Figure GDA0003515564190000101
wherein, USPH(r) is the optical field distribution of an ideal spherical wave, A is the spherical wave amplitude, i is the imaginary unit, k is the wave vector in free space, phi0For the initial phase, r ═ (x, y, z) is the three-dimensional spatial coordinate.
Step 4a, calculating a first light field distribution U of the point light source transmitted to the surface of the sample from the imaging focal plane corresponding to the original fluorescent subimage according to the three-dimensional refractive index distributionL1(ρ) where ρ is (x, y) is the two-dimensional spatial coordinate perpendicular to z, and L is the total number of layers of the sample from the imaging focal plane corresponding to the original fluorescence sub-image to the outer surface of the sample.
Step 5a, according to UL1(p), calculating a first coherent transfer function of the imaging focal plane corresponding to the original fluorescence sub-image
Figure GDA0003515564190000102
And is composed of
Figure GDA0003515564190000103
Updating the first optical transfer function H1(ξ), ξ are two-dimensional coordinates in the frequency domain. Preferably, the first and second electrodes are formed of a metal,
Figure GDA0003515564190000104
: |, represents an auto-correlation calculation. Of course, H1(ξ) may also be obtained in other ways known in the art.
In step 5a
Figure GDA0003515564190000105
Can be obtained by the first method, the second method and other existing methodsObtained by the method.
Method one, according to the focal length of the second objective lens, the U is adjustedL1(rho) is positively propagated to the pupil surface position by utilizing a Fresnel diffraction formula to obtain
Figure GDA0003515564190000106
The second lens is used for receiving ODT mode laser after passing through the sample, laser generated by the SIM super-resolution laser lighting module irradiates the sample after passing through the sample, and original fluorescence of the SIM mode generated by the sample.
Secondly, calculating U by utilizing refocusing back propagation provided by the formula (1)L1(ρ) obtaining
Figure GDA0003515564190000107
Figure GDA0003515564190000108
Figure GDA0003515564190000109
In the formula of UPSF(p) is UL1(ρ) light field distribution propagating back to the imaging focal plane in the opposite direction, i.e. the coherence point spread function, ρLIs the coordinate of any place on the outer surface of the sample, zLFor the distance of the imaging focal plane from the outer surface of the sample, P (-) characterizes the pupil function, UL(ρL) Is the light field distribution of the outer surface of the sample, j is the unit of imaginary number, kmAnd F, is a wave vector of a light field of the illumination laser of the SIM mode in an environment, and Ft { } is Fourier transform.
Further, to
Figure GDA0003515564190000111
The phase of the signal is subjected to phase unwrapping, and decomposition and reconstruction are carried out by utilizing a Zernike polynomial, so that noise interference is removed. The module can calculate the coherent transfer function obtained by different point light sources in the region corresponding to the same original fluorescent imageAnd (4) number averaging, clustering the Zernike polynomial decomposition results obtained by each point source by using a clustering method, or obtaining an average coherent transfer function by using other methods.
In one embodiment, U in said step 4aL1The (ρ) acquisition method specifically includes:
step 41a, setting the point light source as an ideal spherical wave.
Step 42a, calculating a scattering potential f (r) by using a formula (3) according to the three-dimensional refractive index distribution n (r);
Figure GDA0003515564190000112
Figure GDA0003515564190000113
in the formula, nmThe average refractive index of the environment (such as culture solution) where the sample is located can be obtained by an Abbe refractometer, and the refractive index of the cell culture solution used in actual calculation is close to the refractive index of water of 1.333 and is an empirical value; k is a radical ofmWave vector of light field of illumination laser in environment in SIM mode, lambda0The wavelength of the illumination laser in the SIM mode.
Step 43a, according to f (r), calculating UL1(ρ)。
In one embodiment, U in said step 43aL1(ρ) is obtained by using an integral-form Lippmann-schwigger model, and can also be solved by a multilayer beam propagation model provided by the following equations (5) to (7), and can also be obtained by other existing equations:
Ul(ρ，(l+1)Δz)＝Ft-1{Pprop(ξ，Δz)·Ft{Ul(ρ，lΔz)}} (5)
Figure GDA0003515564190000114
Figure GDA0003515564190000115
Figure GDA0003515564190000116
Figure GDA0003515564190000117
in the formula of Ul(ρ, (l +1) Δ z) is the light field distribution, U, for the ideal sinusoidal illumination mode distribution propagating directly from the outer surface of the sample to the l-th layer of the sample without sample effectl(p, l Δ z) is the total light field distribution for the ideal sinusoidal illumination mode distribution to propagate from the outer surface of the sample to the l-1 st layer of the sample,
Figure GDA0003515564190000121
is Ul(ρ, (l) Δ z) the scattered light field, U, generated by the interaction with the sample after passing through the l-th layer of said samplel+1(ρ, (L +1) Δ z) is the total light field distribution of the ideal sinusoidal illumination mode distribution propagating from the outer surface of the sample to the L-th layer of the sample, L1 … …, L, z is the coordinate distribution of the optical axis of the SIM mode, Δ z is the thickness of each layer of the sample, Ft {. cndot } represents the fourier transform, Ft {. cndot-1{. denotes an inverse Fourier transform, Pprep (ξ, Δ z) is an angular spectrum propagation function represented by formula (8), G (ξ, Δ z) is a green function represented by formula (9), fl(p) is the scattering potential of the point light source propagating from the imaging focal plane corresponding to the original fluorescence sub-image to the sample ith layer.
The multilayer beam propagation model is a layer-by-layer propagation module, n takes the total layer number L from 1 to the sample from the imaging focal plane corresponding to the original fluorescence sub-image to the outer surface of the sample, and when L is 1, U is equal to 11(rho, Delta z) is taken as USPH(ρ，Δz)。
Step 6a, according to H1(xi) and corresponding original fluorescence sub-image, and reconstructing again to obtainAnd obtaining a second SIM super-resolution sub-image of the original fluorescence sub-image, wherein the artifact is reduced. The reconstruction method may be a conventional wiener deconvolution method, or may be an iterative optimization method.
And 7a, splicing the second SIM super-resolution sub-image reconstructed in the step 6a and the first SIM super-resolution sub-image corresponding to the region where the original fluorescence sub-image without the fluorescence signal is located into a final complete SIM super-resolution image. In the splicing process, the edge fusion of adjacent sub-images can use a weighted fusion method or other fusion methods to ensure that the overlapped part is smoothly transited from the region corresponding to the previous sub-image to the region corresponding to the next sub-image, thereby ensuring the continuity of the images. For each region without fluorescence signal distribution in the SIM super-resolution image, the corresponding first SIM super-resolution sub-image is directly used, so that the calculation amount can be reduced.
Iterative optimization can also be performed on the final complete SIM super-resolution image output in the step 7a, so that the noise of the super-resolution image is reduced, and the image resolution is increased. The iteration optimization model can add a Hessian regular term and a sparse regular term to jointly modulate the optimization process.
As shown in fig. 1 and 3, an adaptive correction method for a super-resolution microscope image according to an embodiment of the present invention includes:
the method for adaptively correcting a SIM super-resolution image from an illumination path using a three-dimensional refractive index distribution obtained through an original phase image of an ODT modality of a sample includes:
and step 1b, estimating the illumination parameters of each illumination mode according to the original fluorescence image of the SIM mode of the sample. Specifically, the original fluorescence image is multiplied by a phase difference matrix of an illumination mode to carry out phase difference separation, and high-frequency and low-frequency components of the SIM super-resolution image are obtained; then, the correlation between the obtained high-frequency and low-frequency components is calculated, and the correlation of the frequency spectrum is used to estimate the illumination parameters such as the illumination frequency, the illumination phase, the illumination modulation depth coefficient and the like of each illumination direction, wherein the phase difference matrix of the illumination mode is obtained according to the illumination mode used in the microscope hardware design.
And 2b, generating uniform ideal sinusoidal illumination mode distribution at the outer surface of the sample according to the prior information of the illumination mode in sinusoidal distribution and the illumination parameters obtained by the first module.
Step 3b, calculating the total number L of layers from the imaging focal plane corresponding to the original fluorescence image to the outer surface of the sample along the optical axis z axis of the SIM mode and a second light field distribution U of an ideal sine distribution light source transmitted from the outer surface of the sample to the imaging focal plane of the original fluorescence image according to the three-dimensional refractive index distributionL2(ρ) and ρ (x, y) are two-dimensional spatial coordinates perpendicular to the z-axis.
Wherein U in step 3bL2The (ρ) acquisition method specifically includes:
and 31b, calculating the scattering potential f (r) by using the formula (3) according to the three-dimensional refractive index distribution n (r).
Step 32b, according to f (r), calculating UL2(ρ)。UL2The (p) can be obtained by solving using an integral version of the Lippmann-schwigger model, the same multilayer beam propagation models (equations (5) to (7)) as in the above-described embodiment, and other existing methods.
Ul(ρ，(l+1)Δz)＝Ft-1{Pprop(ξ，Δz)·Ft{Ul(ρ，lΔz)}} (5)
Figure GDA0003515564190000131
Figure GDA0003515564190000132
Figure GDA0003515564190000133
Figure GDA0003515564190000134
In the formula，Ul(ρ, (l +1) Δ z) is the light field distribution, U, for the ideal sinusoidal illumination mode distribution propagating directly from the outer surface of the sample to the l-th layer of the sample without sample effectl(p, l Δ z) is the total light field distribution for the ideal sinusoidal illumination mode distribution to propagate from the outer surface of the sample to the l-1 st layer of the sample,
Figure GDA0003515564190000135
is Ul(ρ, (l) Δ z) the scattered light field, U, generated by the interaction with the sample after passing through the l-th layer of said samplel+1(ρ, (L +1) Δ z) is the total light field distribution of the ideal sinusoidal illumination mode distribution propagating from the outer surface of the sample to the L-th layer of the sample, L is the number of layers of the imaging focal plane corresponding to the ideal sinusoidal illumination mode distribution from the original fluorescence image of the SIM modality of the sample, L-1 … …, L, z is the coordinate distribution of the optical axis of the SIM modality, Δ z is the thickness of each layer of the sample, Ft {. cndot } represents the fourier transform, Ft {. cndot-1{. represents an inverse Fourier transform, Pprop(xi, Δ z) is an angular spectrum propagation function represented by formula (8), G (xi, Δ z) is a green function represented by formula (9), fl(p) is the scattering potential of the ideal sinusoidal illumination mode distribution propagating from the imaging focal plane corresponding to the original fluorescence image of the SIM modality of the sample to the ith layer of the sample, when l is 1, U1(ρ, Δ z) takes the ideal sinusoidal illumination pattern generated in step 2 b.
Step 4b, according to UL2(ρ) obtaining an illumination pattern I' (ρ) after scattering distortion due to unevenness of refractive index distribution of the thick sample, which is obtained by equation (11):
I′(ρ)＝|UL2(ρ)|2 (11)。
and 5b, iterating the illumination mode after the scattering distortion by using the optimization model represented by the formula (10), and reconstructing to obtain the optimal SIM super-resolution image
Figure GDA0003515564190000141
The artifacts of (a) are further reduced.
Figure GDA0003515564190000142
Where num is the total number of frames of the original fluorescence image in each period of the illumination pattern, DiFor the i frame of the original fluorescence image, H is a second optical transfer function H2(xi) point spread function, g, obtained after inverse Fourier transformiComprises the following steps: gi(ρ)＝I′i(ρ)·o(ρ)，I′i(ρ) and I'iIs DiCorresponding to the illumination mode after scattering distortion, Hess (o) represents Hessian matrix, | · |. includes1The L1 norm of the matrix is represented, o and o (rho) are SIM super-resolution images to be reconstructed, lambda is a data fidelity term coefficient, the value range of the data fidelity term coefficient is generally between 200 (low signal-to-noise ratio) and 300 (high signal-to-noise ratio), beta is a sparse term constraint coefficient, the value of the sparse term constraint coefficient is generally a multiple relation of 1/20-1/5 of lambda, alpha is a Hessian continuity constraint coefficient, and the value setting principle is as follows: if the data are acquired at intervals, a small value is taken, such as 0.1 or 0.01, if the data are acquired at continuous time, a large value is taken, such as 5, 10 and 20, and if the processed image is too smooth, the value of alpha is reduced.
In the above embodiment, step 5b may also use an optimization model represented by formula (11) instead of formula (10):
Figure GDA0003515564190000143
when the illumination path is adopted alone for the adaptive correction, H in the step 5b2The (xi) can be obtained by constructing an optical transfer function by using an ideal model, or a point spread function H (rho) can be obtained by shooting through fluorescent microbeads, and then H is obtained by utilizing H (rho)2(ξ)，H2(ξ) ═ Ft { h (ρ) }, Ft is the fourier transform.
As shown in fig. 4, the embodiment of the present invention provides an adaptive correction method combining the detection path correction method shown in fig. 2 and the illumination path correction method shown in fig. 3, and corrects both the detection path and the illumination path. The main idea of this combined approach is to simultaneously substitute both the first optical transfer function obtained in the detection path correction method shown in fig. 2 and the distorted illumination pattern obtained in the illumination path correction method shown in fig. 3 into the optimization model represented by equation (10).
Specifically, the adaptive method shown in fig. 4 specifically includes:
and step 1b, estimating the illumination parameters of each illumination mode according to the original fluorescence image of the SIM mode of the sample. Specifically, the original fluorescence image is multiplied by a phase difference matrix of an illumination mode to carry out phase difference separation, and high-frequency and low-frequency components of the SIM super-resolution image are obtained; then, the correlation between the obtained high-frequency and low-frequency components is calculated, and the correlation of the frequency spectrum is used to estimate the illumination parameters such as the illumination frequency, the illumination phase, the illumination modulation depth coefficient and the like of each illumination direction, wherein the phase difference matrix of the illumination mode is obtained according to the illumination mode used in the microscope hardware design.
And 2b, generating uniform ideal sinusoidal illumination mode distribution at the outer surface of the sample according to the prior information of the illumination mode in sinusoidal distribution and the illumination parameters obtained by the first module.
Step 3b, calculating the total number L of layers from the imaging focal plane corresponding to the original fluorescence image to the outer surface of the sample along the optical axis z axis of the SIM mode and a second light field distribution U of an ideal sine distribution light source transmitted from the outer surface of the sample to the imaging focal plane of the original fluorescence image according to the three-dimensional refractive index distributionL2(ρ) and ρ are two-dimensional spatial coordinates perpendicular to (x, y).
Wherein U in step 3bL2The (ρ) acquisition method specifically includes:
and 31b, calculating the scattering potential f (r) by using the formula (3) according to the three-dimensional refractive index distribution n (r).
Step 32b, according to f (r), calculating UL2(ρ)。UL2The (p) can be obtained by solving using an integral version of the Lippmann-schwigger model, using a multilayer beam propagation model provided by the following equations (5) to (7), and using other existing methods.
Ul(ρ，(l+1)Δz)＝Ft-1{Pprop(ξ，Δz)·Ft{Ul(ρ，lΔz)}} (5)
Figure GDA0003515564190000151
Figure GDA0003515564190000152
Figure GDA0003515564190000153
Figure GDA0003515564190000154
In the formula of Ul(ρ, (l +1) Δ z) is the light field distribution, U, for the ideal sinusoidal illumination mode distribution propagating directly from the outer surface of the sample to the l-th layer of the sample without sample effectl(p, l Δ z) is the total light field distribution for the ideal sinusoidal illumination mode distribution to propagate from the outer surface of the sample to the l-1 st layer of the sample,
Figure GDA0003515564190000155
is Ul(ρ, (l) Δ z) the scattered light field, U, generated by the interaction with the sample after passing through the l-th layer of said samplel+1(ρ, (L +1) Δ z) is the total light field distribution of the ideal sinusoidal illumination mode distribution propagating from the outer surface of the sample to the L-th layer of the sample, L is the number of layers of the imaging focal plane corresponding to the ideal sinusoidal illumination mode distribution from the original fluorescence image of the SIM modality of the sample, L-1 … …, L, z is the coordinate distribution of the optical axis of the SIM modality, Δ z is the thickness of each layer of the sample, Ft {. cndot } represents the fourier transform, Ft {. cndot-1{. represents an inverse Fourier transform, Pprop(xi, Δ z) is an angular spectrum propagation function represented by formula (8), G (xi, Δ z) is a green function represented by formula (9), fl(p) is the distribution of the ideal sinusoidal illumination patternThe scattering potential of the I layer of the sample is propagated to the imaging focal plane corresponding to the original fluorescence image of the SIM mode of the sample, and when l is 1, U is1(ρ, Δ z) takes the ideal sinusoidal illumination pattern generated in step 2 b.
Step 4b, according to UL2(ρ) obtaining an illumination pattern I' (ρ) after scattering distortion due to unevenness of refractive index distribution of the thick sample, which is obtained by equation (12):
I′(ρ)＝|UL2(ρ)|2 (12)。
in formula (12), I'i(ρ) and I'iIs DiThe corresponding scattered distorted illumination pattern, and I' (ρ) here refers to the unspecified illumination pattern.
And 5b, iterating the illumination mode after the scattering distortion by using the optimization model represented by the formula (10), and reconstructing to obtain the optimal SIM super-resolution image
Figure GDA0003515564190000161
The artifacts of (a) are further reduced.
Figure GDA0003515564190000162
Where num is the total number of frames of the original fluorescence image in each period of the illumination pattern, DiFor the i frame of the original fluorescence image, H is a second optical transfer function H2(xi) point spread function, g, obtained after inverse Fourier transformiComprises the following steps: gi(ρ)＝I′i(ρ)·o(ρ)，I′i(ρ) and I'iIs DiCorresponding to the illumination mode after scattering distortion, Hess (o) represents Hessian matrix, | · |. includes1The L1 norm of the matrix is represented, o and o (rho) are SIM super-resolution images to be reconstructed, lambda is a data fidelity term coefficient, the value range of the data fidelity term coefficient is generally between 200 (low signal-to-noise ratio) and 300 (high signal-to-noise ratio), beta is a sparse term constraint coefficient, the value of the sparse term constraint coefficient is generally a multiple relation of 1/20-1/5 of lambda, alpha is a Hessian continuity constraint coefficient, and the value setting principle is as follows: if the data are acquired at intervals, the data are comparedSmall values, such as 0.1 or 0.01, if the data are collected continuously, larger values, such as 5, 10 and 20, are taken, and if the processed image is too smooth, the value of alpha is reduced.
H in said step 5b2(ξ) is obtained by:
step 51b, the three-dimensional refractive index profile obtained by the original phase image of the ODT modality of the sample is utilized.
And step 52b, reconstructing and obtaining an initial SIM super-resolution image with an artifact according to the optical transfer function and the original fluorescence image of the SIM mode of the sample.
And 53b, dividing the original fluorescent image into a plurality of original fluorescent sub-images according to the first division mode, wherein the edges of the adjacent sub-images are overlapped with each other.
And step 54b, finding out an imaging focal plane corresponding to each original fluorescence sub-image containing the fluorescence signal and the number of sample layers corresponding to the imaging focal plane along the optical axis z axis of the SIM mode through the three-dimensional refractive index distribution, and arranging a point light source at the center of each original fluorescence sub-image containing the fluorescence signal and the selected point of the adjacent region of the center.
Step 55b, calculating a first light field distribution U of the point light source transmitted to the surface of the sample from the imaging focal plane corresponding to the original fluorescent sub-image according to the three-dimensional refractive index distributionL1(ρ) where ρ is (x, y) is the two-dimensional spatial coordinate perpendicular to z, and L is the total number of layers of the sample from the imaging focal plane corresponding to the original fluorescence sub-image to the outer surface of the sample.
Step 56b, according to UL1(p), calculating a first coherent transfer function of the imaging focal plane corresponding to the original fluorescence sub-image
Figure GDA0003515564190000171
And is composed of
Figure GDA0003515564190000172
Updating the second optical transfer function H2(ξ)，
Figure GDA0003515564190000173
Figure GDA0003515564190000174
ξ is a two-dimensional coordinate in the frequency domain, which represents an autocorrelation calculation.
According to the invention, by utilizing ODT modal shooting in a dual-mode microscope system to obtain sample refractive index information and a wave optical propagation theory, the distortion of an illumination mode (attenuation pattern) at each position of an illumination path of the super-resolution microscope and an optical transfer function at each position of a detection path are corrected in a self-adaptive manner, and then the illumination mode and the optical transfer function information after the distortion correction are introduced into a reconstruction model, a more accurate image reconstruction method is designed.
Finally, it should be pointed out that: the above examples are only for illustrating the technical solutions of the present invention, and are not limited thereto. Those of ordinary skill in the art will understand that: modifications can be made to the technical solutions described in the foregoing embodiments, or some technical features may be equivalently replaced; such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims (10)

1. An adaptive correction method for super-resolution microscope images is characterized by comprising the following steps:
the method for self-adaptive correction of SIM super-resolution image from imaging detection path by using three-dimensional refractive index distribution obtained by original phase image of ODT mode of sample includes:
step 1a, reconstructing to obtain an initial SIM super-resolution image with an artifact according to an optical transfer function and an original fluorescence image of the sample in an SIM mode;
step 2a, dividing the initial SIM super-resolution image into a plurality of first SIM super-resolution sub-images according to a first dividing mode, dividing the original fluorescent image into a plurality of original fluorescent sub-images according to the first dividing mode, and overlapping the edges of the adjacent sub-images;
step 3a, finding out an imaging focal plane corresponding to each original fluorescence sub-image containing a fluorescence signal and a sample layer number corresponding to the imaging focal plane along an optical axis z axis of the SIM mode through the three-dimensional refractive index distribution, and arranging a point light source at the center of each original fluorescence sub-image containing the fluorescence signal and a selected point of an adjacent region of the center;
step 4a, calculating a first light field distribution U of the point light source transmitted to the surface of the sample from the imaging focal plane corresponding to the original fluorescent subimage according to the three-dimensional refractive index distributionL1(ρ), ρ ═ (x, y) is the two-dimensional spatial coordinate perpendicular to z, and L is the total number of layers of the sample from the imaging focal plane corresponding to the original fluorescence sub-image to the outer surface of the sample;
step 5a, according to UL1(p), calculating a first coherent transfer function of the imaging focal plane corresponding to the original fluorescence sub-image
Figure FDA0003515564180000011
And is composed of
Figure FDA0003515564180000012
Updating the first optical transfer function H1(ξ), ξ being a two-dimensional coordinate in the frequency domain;
step 6a, according to H1(ξ) and the corresponding original fluorescence sub-image, and reconstructing again to obtain a second SIM super-resolution sub-image of the original fluorescence sub-image;
and 7a, splicing the second SIM super-resolution sub-image reconstructed in the step 6a and the first SIM super-resolution sub-image corresponding to the region where the original fluorescence sub-image without the fluorescence signal is located into a final complete SIM super-resolution image.
2. The adaptive correction method for super resolution microscope images according to claim 1, wherein the step 5a is performed by
Figure FDA0003515564180000013
Obtained by one of the following two methods:
method one, according to the focal length of the second objective lens, the U is adjustedL1(rho) is positively propagated to the pupil surface position by utilizing a Fresnel diffraction formula to obtain
Figure FDA0003515564180000014
The second lens is used for receiving ODT modal laser after passing through the sample, irradiating the sample after the laser generated by the SIM super-resolution laser lighting module passes through the sample, and generating original fluorescence in an SIM mode generated by the sample; wherein the second lens is used for receiving ODT mode laser after passing through the sample, original fluorescence of SIM mode generated by the sample and allowing the SIM mode laser to pass through and irradiate the sample;
secondly, calculating U by utilizing refocusing back propagation provided by the formula (1)L1(ρ) obtaining
Figure FDA0003515564180000021
Figure FDA0003515564180000022
Figure FDA0003515564180000023
In the formula of UPSF(p) is UL1(ρ) light field distribution propagating back to the imaging focal plane in the opposite direction, i.e. the coherence point spread function, ρLIs the coordinate of any place on the outer surface of the sample, zLCharacterizing the pupil function, U, for the physical distance of the imaging focal plane to the outer surface of the sample, P (-), P (-. cndot.)L(ρL) Is the light field distribution of the outer surface of the sample, j is the unit of imaginary number, kmFor the wave vector of the light field of the illumination laser of the SIM mode in the environment, Ft { } is Fourier transform, lambda0Is a stand forThe wavelength of the illumination laser in the SIM mode.
3. The adaptive correction method for super resolution microscope images according to claim 2, wherein U in step 4a is set to zeroL1The (ρ) acquisition method specifically includes:
step 41a, setting the point light source as an ideal spherical wave;
step 42a, calculating a scattering potential f (r) by using a formula (3) according to the three-dimensional refractive index distribution n (r);
Figure FDA0003515564180000024
Figure FDA0003515564180000025
in the formula, nmIs the average refractive index, k, of the environment in which the sample is locatedmA wave vector of a light field of the illumination laser in an environment, which is a SIM mode;
step 43a, according to f (r), calculating UL1(ρ)。
4. The adaptive correction method for super resolution microscope images according to claim 3, wherein U in step 43a is set toL1(ρ) is solved by using a Lippmann-schwigger model in an integral form, or using a multilayer beam propagation model provided by the following equations (5) to (7):
Ul(ρ，(l+1)Δz)＝Ft-1{Pprop(ξ，Δz)·Ft{Ul(ρ，lΔz)}} (5)
Figure FDA0003515564180000031
Figure FDA0003515564180000032
Figure FDA0003515564180000033
Figure FDA0003515564180000034
in the formula of Ul(ρ, (l +1) Δ z) is the light field distribution, U, of the point light source propagating directly from the imaging focal plane corresponding to the original fluorescence sub-image to the ith layer of the sample without sample actionl(p, l Δ z) is the total light field distribution of the point light source propagating from the corresponding imaging focal plane of the original fluorescence sub-image to layer l-1 of the sample,
Figure FDA0003515564180000035
is Ul(ρ, (l) Δ z) the scattered light field, U, generated by the interaction with the sample after passing through the l-th layer of said samplel+1(ρ, (L +1) Δ z) is the total light field distribution of the point light source propagating from the imaging focal plane corresponding to the original fluorescence sub-image to the L-th layer of the sample, L is the number of layers of the point light source from the imaging focal plane corresponding to the original fluorescence sub-image to the outer surface of the sample, L is 1 … …, L, z is the coordinate distribution of the optical axis of the SIM-mode, Δ z is the thickness of each layer of the sample, Ft {. cndot.) represents the fourier transform, Ft {. cndot-1{. represents an inverse Fourier transform, Pprop(xi, Δ z) is an angular spectrum propagation function represented by formula (8), G (xi, Δ z) is a green function represented by formula (9), fl(p) is the scattering potential of the point light source propagating from the imaging focal plane corresponding to the original fluorescent sub-image to the ith layer of the sample, when l is 1, U is1(rho, Delta z) is taken as USPH(ρ，Δz)。
5. An adaptive correction method for super-resolution microscope images is characterized by comprising the following steps:
the method for adaptively correcting a SIM super-resolution image from an illumination path using a three-dimensional refractive index distribution obtained through an original phase image of an ODT modality of a sample includes:
step 1b, estimating and solving illumination parameters of each illumination mode according to the original fluorescence image of the SIM mode of the sample;
step 2b, generating uniform ideal sinusoidal illumination mode distribution at the outer surface of the sample according to the prior information of the illumination mode in sinusoidal distribution and the illumination parameters obtained by the first module;
step 3b, calculating the total number L of layers from the imaging focal plane corresponding to the original fluorescence image to the outer surface of the sample along the optical axis z axis of the SIM mode and the second light field distribution U of the ideal sine distribution light source transmitted from the outer surface of the sample to the imaging focal plane of the original fluorescence image according to the three-dimensional refractive index distributionL2(ρ) where ρ is (x, y) a two-dimensional spatial coordinate perpendicular to the z-axis;
step 4b, according to UL2(ρ) obtaining the illumination pattern after scattering distortion;
and 5b, iterating the illumination mode after the scattering distortion by using the optimization model represented by the formula (10), and reconstructing to obtain the optimal SIM super-resolution image
Figure FDA0003515564180000041
Figure FDA0003515564180000042
Where num is the total number of frames of the original fluorescence image in each period of the illumination pattern, DiFor the i frame of the original fluorescence image, H is a second optical transfer function H2(xi) point spread function, g, obtained after inverse Fourier transformiComprises the following steps: gi(ρ)＝I′i(ρ)·o(ρ)，I′i(ρ) and I'iIs DiCorresponding to the illumination mode after scattering distortion, Hess (o) represents Hessian matrix, | · |. includes1L1 norm representing the matrix, o and o (ρ) to be weightedIn the built SIM super-resolution image, lambda is a data fidelity term coefficient, alpha is a Hessian continuity constraint coefficient, beta is a sparse term constraint coefficient, and xi is a two-dimensional coordinate in a frequency domain.
6. The adaptive correction method for super resolution microscope images according to claim 5, wherein U in step 3bL2The (ρ) acquisition method specifically includes:
step 31b, calculating a scattering potential f (r) by using the formula (3) according to the three-dimensional refractive index distribution n (r);
Figure FDA0003515564180000043
Figure FDA0003515564180000044
in the formula, nmIs the average refractive index, k, of the environment in which the sample is locatedmWave vector of light field of illumination laser in environment in SIM mode, lambda0A wavelength of an illumination laser in the SIM mode;
step 32b, according to f (r), calculating UL2(ρ)。
7. The adaptive correction method for super resolution microscope images according to claim 6, wherein U in step 32bL2(ρ) is solved by using a Lippmann-schwigger model in an integral form or using a multilayer beam propagation model provided by the following equations (5) to (7):
Ul(ρ，(l+1)Δz)＝Ft-1{Pprop(ξ，Δz)·Ft{Ul(ρ，lΔz)}} (5)
Figure FDA0003515564180000045
Figure FDA0003515564180000046
Figure FDA0003515564180000051
Figure FDA0003515564180000052
in the formula of Ul(ρ, (l +1) Δ z) is the light field distribution, U, for the ideal sinusoidal illumination mode distribution propagating directly from the outer surface of the sample to the l-th layer of the sample without sample effectl(p, l Δ z) is the total light field distribution for the ideal sinusoidal illumination mode distribution to propagate from the outer surface of the sample to the l-1 st layer of the sample,
Figure FDA0003515564180000053
is Ul(ρ, (l) Δ z) the scattered light field, U, generated by the interaction with the sample after passing through the l-th layer of said samplel+1(ρ, (L +1) Δ z) is the total light field distribution of the ideal sinusoidal illumination mode distribution propagating from the outer surface of the sample to the L-th layer of the sample, L is the number of layers of the imaging focal plane corresponding to the ideal sinusoidal illumination mode distribution from the original fluorescence image of the SIM modality of the sample, L-1 … …, L, z is the coordinate distribution of the optical axis of the SIM modality, Δ z is the thickness of each layer of the sample, Ft {. cndot } represents the fourier transform, Ft {. cndot-1{. represents an inverse Fourier transform, Pprop(xi, Δ z) is an angular spectrum propagation function represented by formula (8), G (xi, Δ z) is a green function represented by formula (9), fl(p) is the scattering potential of the ideal sinusoidal illumination mode distribution propagating from the imaging focal plane corresponding to the original fluorescence image of the SIM modality of the sample to the ith layer of the sample, when l is 1, U1(ρ, Δ z) takes the ideal sinusoidal illumination pattern generated in step 2 b.
8. Such as rightThe method for adaptive correction of super resolution microscope images as set forth in any one of claims 5 to 7, wherein H in step 5b2(xi) is obtained by one of two ways:
the method comprises the steps of firstly, constructing an optical transfer function by using an ideal model to obtain;
and secondly, acquiring a point spread function H (rho) by fluorescent bead shooting, and acquiring H by using the H (rho)2(ξ)，H2(ξ) ═ Ft { h (ρ) }, Ft is the fourier transform.
9. The adaptive correction method for super resolution microscope images according to claim 5, wherein H in step 5b2(ξ) is obtained by:
step 51b, using a three-dimensional refractive index profile obtained by an original phase image of an ODT mode of a sample;
step 52b, reconstructing to obtain an initial SIM super-resolution image with an artifact according to the optical transfer function and the original fluorescence image of the sample in the SIM mode;
step 53b, dividing the original fluorescent image into a plurality of original fluorescent sub-images according to a first dividing mode, wherein the edges of the adjacent sub-images are overlapped with each other;
step 54b, finding out an imaging focal plane corresponding to each original fluorescence sub-image containing the fluorescence signal and the number of sample layers corresponding to the imaging focal plane along the optical axis z axis of the SIM mode through the three-dimensional refractive index distribution, and arranging a point light source at the center of each original fluorescence sub-image containing the fluorescence signal and a selected point of an adjacent region of the center;
step 55b, calculating a first light field distribution U of the point light source transmitted to the surface of the sample from the imaging focal plane corresponding to the original fluorescent sub-image according to the three-dimensional refractive index distributionL1(ρ), ρ ═ (x, y) is the two-dimensional spatial coordinate perpendicular to z, and L is the total number of layers of the sample from the imaging focal plane corresponding to the original fluorescence sub-image to the outer surface of the sample;
step 56b, according to UL1(p) calculating the original fluorescence photon image pairFirst coherent transfer function of the imaging focal plane
Figure FDA0003515564180000061
And is composed of
Figure FDA0003515564180000062
Updating the second optical transfer function H2(ξ)，
Figure FDA0003515564180000063
Figure FDA0003515564180000064
ξ is a two-dimensional coordinate in the frequency domain, which represents an autocorrelation calculation.
10. A SIM-ODT bimodal system, comprising:
an ODT lighting module for generating ODT mode laser light;
the SIM super-resolution laser lighting module is used for generating SIM modal laser in sinusoidal distribution;
the first objective lens is arranged between one side of the sample and the light outlet of the ODT lighting module and is used for irradiating the sample by the laser generated by the ODT lighting module;
the second objective is arranged on the other side of the sample and is used for receiving ODT (optical time difference) mode laser after passing through the sample, passing the SIM mode laser and irradiating the sample, and passing original fluorescence of a SIM mode generated by the sample;
the SIM super-resolution image acquisition module is used for acquiring an original fluorescence image of the SIM modality;
the ODT image acquisition module is used for acquiring an original phase image of an ODT modality;
the sample three-dimensional refractive index distribution calculation module is used for processing the original phase image by utilizing a reconstruction algorithm of diffraction tomography, and reconstructing to obtain the three-dimensional refractive index distribution of the unmarked sample;
an adaptive image correction module for adaptively correcting the optical transfer function and/or the distorted illumination pattern of the SIM modality with the adaptive correction method of super resolution microscope images as claimed in claim 1 or 5 or 9;
the 2D/3D-SIM super-resolution image reconstruction module is used for reconstructing the original fluorescence image and the optical transfer function according to the optical transfer function and/or the distorted illumination mode corrected by the self-adaptive image correction module to obtain a self-adaptive corrected SIM super-resolution image;
and the bimodal image fusion module is used for fusing the SIM super-resolution image subjected to self-adaptive correction with the three-dimensional refractive index distribution according to a spatial position relationship, and simultaneously carrying out three-dimensional high-resolution dynamic imaging and deep specific two-dimensional or three-dimensional super-resolution dynamic imaging on the sample.
CN202110386643.4A 2021-04-12 2021-04-12 Self-adaptive correction method of super-resolution microscope image and SIM-ODT (subscriber identity module-ODT) bimodal system Active CN113112405B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110386643.4A CN113112405B (en) 2021-04-12 2021-04-12 Self-adaptive correction method of super-resolution microscope image and SIM-ODT (subscriber identity module-ODT) bimodal system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110386643.4A CN113112405B (en) 2021-04-12 2021-04-12 Self-adaptive correction method of super-resolution microscope image and SIM-ODT (subscriber identity module-ODT) bimodal system

Publications (2)

Publication Number Publication Date
CN113112405A CN113112405A (en) 2021-07-13
CN113112405B true CN113112405B (en) 2022-04-12

Family

ID=76715406

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110386643.4A Active CN113112405B (en) 2021-04-12 2021-04-12 Self-adaptive correction method of super-resolution microscope image and SIM-ODT (subscriber identity module-ODT) bimodal system

Country Status (1)

Country Link
CN (1) CN113112405B (en)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114092331B (en) * 2021-11-24 2024-05-24 北京大学 Structured light super-resolution reconstruction method combined with phase-locked amplification pretreatment
CN115100033B (en) * 2022-05-20 2023-09-08 浙江大学 Fluorescent microscopic image super-resolution reconstruction method and device and computing equipment
CA3210853A1 (en) * 2022-09-26 2024-03-26 Weisong ZHAO Super-resolution microscopic imaging method and apparatus, computer device, and storage medium

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105300941A (en) * 2015-10-20 2016-02-03 北京大学 Fluorescent optical sheet microscopic imaging system and method
CN109035143A (en) * 2018-07-17 2018-12-18 华中科技大学 A kind of three-dimensional super-resolution method based on the imaging of Bezier mating plate
US10169852B1 (en) * 2018-07-03 2019-01-01 Nanotronics Imaging, Inc. Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging
CN109712072A (en) * 2018-12-15 2019-05-03 浙江大学 Striped illumination Fourier iteration based on total internal reflection updates super-resolution micro imaging method
CN111077121A (en) * 2019-12-06 2020-04-28 中国科学院西安光学精密机械研究所 Rapid method and system for directly reconstructing structured light illumination super-resolution image in space domain
CN111127320A (en) * 2019-12-23 2020-05-08 哈尔滨工业大学（威海） Photoacoustic image super-resolution reconstruction method and device based on deep learning
CN111145089A (en) * 2019-11-18 2020-05-12 中国科学院苏州生物医学工程技术研究所 High fidelity image reconstruction method, system, computer equipment and storage medium

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104515759B (en) * 2014-12-16 2017-05-31 中国科学院苏州生物医学工程技术研究所 Nonlinear organization optical illumination micro imaging method and system

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105300941A (en) * 2015-10-20 2016-02-03 北京大学 Fluorescent optical sheet microscopic imaging system and method
US10169852B1 (en) * 2018-07-03 2019-01-01 Nanotronics Imaging, Inc. Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging
CN109035143A (en) * 2018-07-17 2018-12-18 华中科技大学 A kind of three-dimensional super-resolution method based on the imaging of Bezier mating plate
CN109712072A (en) * 2018-12-15 2019-05-03 浙江大学 Striped illumination Fourier iteration based on total internal reflection updates super-resolution micro imaging method
CN111145089A (en) * 2019-11-18 2020-05-12 中国科学院苏州生物医学工程技术研究所 High fidelity image reconstruction method, system, computer equipment and storage medium
CN111077121A (en) * 2019-12-06 2020-04-28 中国科学院西安光学精密机械研究所 Rapid method and system for directly reconstructing structured light illumination super-resolution image in space domain
CN111127320A (en) * 2019-12-23 2020-05-08 哈尔滨工业大学（威海） Photoacoustic image super-resolution reconstruction method and device based on deep learning

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
Structured illumination microscopy artifacts caused by illumination scattering;Mo Yanquan,et al.;《ResearchGate》;20210131;第1-18页 *
Super-resolution fluorescence-assisted diffraction computational tomography reveals the three-dimensional landscape of the celluar organelle interactome;Dashan Dong，et al.;《Light:Science&Applications》;20201231;第9卷(第1期);第1-15页 *
自适应光学在生物荧光显微成像技术中的应用;张财华等;《中国科学》;20170831;第47卷(第8期);第1-14页 *

Also Published As

Publication number Publication date
CN113112405A (en) 2021-07-13

Similar Documents

Publication Publication Date Title
CN113112405B (en) Self-adaptive correction method of super-resolution microscope image and SIM-ODT (subscriber identity module-ODT) bimodal system
Kuś et al. Holographic tomography: hardware and software solutions for 3D quantitative biomedical imaging
US8731272B2 (en) Computational adaptive optics for interferometric synthetic aperture microscopy and other interferometric imaging
US20200234080A1 (en) Reducing noise in an image
Karpinsky et al. High-resolution, real-time 3D imaging with fringe analysis
Ralston et al. Deconvolution methods for mitigation of transverse blurring in optical coherence tomography
CN111292386B (en) CT projection metal trace complement metal artifact correction method based on U-net
JP6112872B2 (en) Imaging system, image processing method, and imaging apparatus
Nadeau et al. Multifrequency synthesis and extraction using square wave projection patterns for quantitative tissue imaging
US10908060B2 (en) Method for determining parameters of a particle
Carpio et al. Noninvasive imaging of three-dimensional micro and nanostructures by topological methods
US9052180B2 (en) Spatial light interference tomography
Lee et al. High-throughput volumetric adaptive optical imaging using compressed time-reversal matrix
Bai et al. Compressed blind deconvolution and denoising for complementary beam subtraction light-sheet fluorescence microscopy
Kang et al. Tracing multiple scattering trajectories for deep optical imaging in scattering media
Kumar et al. Digital adaptive optics based on digital lateral shearing of the computed pupil field for point scanning retinal swept source OCT
Zuo et al. Spectral crosstalk in photoacoustic computed tomography
Wang et al. Approximate back‐projection method for improving lateral resolution in circular‐scanning‐based photoacoustic tomography
Kang et al. Mapping nanoscale topographic features in thick tissues with speckle diffraction tomography
Tang et al. Learning spatially variant degradation for unsupervised blind photoacoustic tomography image restoration
Obando-Vásquez et al. Semi-heuristic phase compensation in digital holographic microscopy for stable and accurate quantitative phase imaging of moving objects
JP7228189B2 (en) Method and apparatus for evaluating cytotoxicity
Fridberger et al. Measuring hearing organ vibration patterns with confocal microscopy and optical flow
Deng et al. Multi-segmented feature coupling for jointly reconstructing initial pressure and speed of sound in photoacoustic computed tomography
Ma et al. Spatial weight matrix in dimensionality reduction reconstruction for micro-electromechanical system-based photoacoustic microscopy

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant