CN113112405A - Self-adaptive correction method of super-resolution microscope image and SIM-ODT (subscriber identity module-ODT) bimodal system - Google Patents

Self-adaptive correction method of super-resolution microscope image and SIM-ODT (subscriber identity module-ODT) bimodal system Download PDF

Info

Publication number
CN113112405A
CN113112405A CN202110386643.4A CN202110386643A CN113112405A CN 113112405 A CN113112405 A CN 113112405A CN 202110386643 A CN202110386643 A CN 202110386643A CN 113112405 A CN113112405 A CN 113112405A
Authority
CN
China
Prior art keywords
sample
image
sim
super
resolution
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110386643.4A
Other languages
Chinese (zh)
Other versions
CN113112405B (en
Inventor
莫燕权
陈良怡
毛珩
范骏超
丰帆
梁林涛
屠锐
杜珂
杨宏润
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangzhou Chaoshiji Biotechnology Co ltd
Original Assignee
Guangzhou Chaoshiji Biotechnology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangzhou Chaoshiji Biotechnology Co ltd filed Critical Guangzhou Chaoshiji Biotechnology Co ltd
Priority to CN202110386643.4A priority Critical patent/CN113112405B/en
Publication of CN113112405A publication Critical patent/CN113112405A/en
Application granted granted Critical
Publication of CN113112405B publication Critical patent/CN113112405B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformation in the plane of the image
    • G06T3/40—Scaling the whole image or part thereof
    • G06T3/4053—Super resolution, i.e. output image resolution higher than sensor resolution
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/58—Optics for apodization or superresolution; Optical synthetic aperture systems

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Optics & Photonics (AREA)
  • Microscoopes, Condenser (AREA)
  • Investigating, Analyzing Materials By Fluorescence Or Luminescence (AREA)

Abstract

The invention discloses a self-adaptive correction method of a super-resolution microscope image and an SIM-ODT bimodal system, wherein the method comprises the following steps: step 1a, reconstructing to obtain an initial SIM super-resolution image with an artifact according to an optical transfer function and an original fluorescence image of a sample in an SIM mode; step 2a, dividing the initial SIM super-resolution image and the original fluorescence image into a plurality of first SIM super-resolution sub-images and original fluorescence sub-images according to a first dividing mode; step 3a, finding out an imaging focal plane corresponding to each original fluorescent sub-image containing the fluorescent signal and a sample layer number corresponding to the imaging focal plane along an optical axis of an SIM mode through three-dimensional refractive index distribution, and arranging a point light source at the center of each original fluorescent sub-image containing the fluorescent signal and a selected point of an adjacent region of the center; step 4a, calculating first light field distribution of the point light source transmitted to the surface of the sample from the imaging focal plane corresponding to the original fluorescent sub-image; step 5a, updating the first optical transfer function; and 6a, reconstructing to obtain a second SIM super-resolution sub-image of the original fluorescence sub-image. The invention can carry out self-adaptive correction on all aberrations and artificial artifacts of the super-resolution microscope reconstructed image.

Description

Self-adaptive correction method of super-resolution microscope image and SIM-ODT (subscriber identity module-ODT) bimodal system
Technical Field
The invention relates to the technical field of super-resolution microscopic imaging, in particular to a self-adaptive correction method of a super-resolution microscope image and an SIM-ODT (subscriber identity module-ODT) bimodal system.
Background
The SIM (structured light microscope) breaks through the optical diffraction limit by adopting a sinusoidal illumination mode with multiple directions and multiple phases for spectral modulation, so as to achieve the purpose of super-resolution (super-resolution) imaging. Meanwhile, compared with other super-resolution technologies such as an stimulated radiation depletion microscope STED and a single-molecule positioning microscope PALM/STROM, the SIM has the advantages of small sample specificity requirement, small photon number, small phototoxicity, high imaging speed and the like, and becomes one of the most widely applied super-resolution technologies in living cell research. However, despite its many advantages, SIM is susceptible to reconstruction artifacts: inaccurate parameter estimation, low image signal-to-noise ratio, large optical transfer function error, non-negligible background fluorescence of an out-of-focus plane and the like all bring difficulty to reconstruction of the SIM super-resolution image. For a sample with a large thickness, the refractive index distribution inside the sample is very different because the distribution of organelles is not uniform. These structural differences cause scattering of the SIM illumination laser and the excited fluorescence, which in turn causes imaging aberrations to the microscope, which is also the most important factor limiting the imaging depth of the SIM, so that the current SIM is still limited to imaging in the range of thin sample or cell shallow region.
If the problem of artifacts of deep super-resolution reconstructed images of SIM thick samples can be solved, the application range of the SIM can be greatly expanded, and the experimental research progress of cell biology is facilitated. In a conventional SIM super-resolution image reconstruction algorithm, an Optical Transfer Function (OTF) used for reconstruction is basically derived from an OTF constructed by an ideal physical model or an OTF obtained by shooting with fluorescent beads, and is an OTF which is too ideal and unmatched for an actual biological sample. In the conventional method for solving the super-resolution image artifact, there is a method for directly adding a regular term in an image reconstruction algorithm model to further constrain the process, and there is also a method for detecting aberration by adding a wavefront sensor in an optical path and then correcting the wavefront aberration by using a deformable mirror by using an adaptive optics means, or directly correcting the aberration in the optical path by using a method without wavefront sensing. However, the method of adding a regularization term to the reconstruction model requires that the original fluorescence image of the SIM modality acquired by the microscope and the parameters used for reconstruction, such as OTF, are indistinguishable, which is obviously difficult to achieve for thick samples. However, the method of adaptive optical correction can only correct global aberration, but cannot correct local aberration.
Disclosure of Invention
The invention aims to provide a self-adaptive correction method of a super-resolution microscope image and an SIM-ODT dual-mode system, which can carry out self-adaptive correction on various aberrations and artificial artifacts of a super-resolution microscope reconstructed image.
In order to achieve the above object, the present invention provides an adaptive correction method for a super-resolution microscope image, including:
the method of adaptively correcting a SIM super-resolution image from an imaging probe path using a three-dimensional refractive index distribution obtained through an original phase image of an ODT (optical diffraction tomography) mode of a sample includes:
step 1a, reconstructing to obtain an initial SIM super-resolution image with an artifact according to an optical transfer function and an original fluorescence image of the sample in an SIM mode;
step 2a, dividing the initial SIM super-resolution image into a plurality of first SIM super-resolution sub-images according to a first dividing mode, dividing the original fluorescent image into a plurality of original fluorescent sub-images according to the first dividing mode, and overlapping the edges of the adjacent sub-images;
step 3a, finding out an imaging focal plane corresponding to each original fluorescence sub-image containing a fluorescence signal and a sample layer number corresponding to the imaging focal plane along an optical axis z axis of the SIM mode through the three-dimensional refractive index distribution, and arranging a point light source at the center of each original fluorescence sub-image containing the fluorescence signal and a selected point of an adjacent region of the center;
step 4a, calculating a first light field distribution U of the point light source transmitted to the surface of the sample from the imaging focal plane corresponding to the original fluorescent subimage according to the three-dimensional refractive index distributionL1(ρ), ρ ═ (x, y) is the two-dimensional spatial coordinate perpendicular to z, and L is the total number of layers of the sample from the imaging focal plane corresponding to the original fluorescence sub-image to the outer surface of the sample;
step 5a, according to UL1(p), calculating a first coherent transfer function of the imaging focal plane corresponding to the original fluorescence sub-image
Figure BDA0003015321130000021
And is composed of
Figure BDA0003015321130000022
Updating the first optical transfer function H1(ξ), ξ being a two-dimensional coordinate in the frequency domain;
step 6a, according to H1(xi) and the corresponding original fluorescent sub-image, and reconstructing again to obtain a second SIM super-resolution sub-image of the original fluorescent sub-imageAn image;
and 7a, splicing the second SIM super-resolution sub-image reconstructed in the step 6a and the first SIM super-resolution sub-image corresponding to the region where the original fluorescence sub-image without the fluorescence signal is located into a final complete SIM super-resolution image.
Further, in said step 5a
Figure BDA0003015321130000031
Obtained by one of the following two methods:
method one, according to the focal length of the second objective lens, the U is adjustedL1(rho) is positively propagated to the pupil surface position by utilizing a Fresnel diffraction formula to obtain
Figure BDA0003015321130000032
The second lens is used for receiving ODT modal laser after passing through the sample, irradiating the sample after the laser generated by the SIM super-resolution laser lighting module passes through the sample, and generating original fluorescence in an SIM mode generated by the sample; wherein the second lens is used for receiving ODT mode laser after passing through the sample, original fluorescence of SIM mode generated by the sample and allowing the SIM mode laser to pass through and irradiate the sample;
secondly, calculating U by utilizing refocusing back propagation provided by the formula (1)L1(ρ) obtaining
Figure BDA0003015321130000033
Figure BDA0003015321130000034
Figure BDA0003015321130000035
In the formula of UPSF(p) is UL1(ρ) light field distribution propagating back to the imaging focal plane in the opposite direction, i.e. the coherence point spread function, ρLIs the coordinate anywhere on the outer surface of the specimen,zLcharacterizing the pupil function, U, for the physical distance of the imaging focal plane to the outer surface of the sample, P (-), P (-. cndot.)L(ρL) Is the light field distribution of the outer surface of the sample, j is the unit of imaginary number, kmFt {. is the Fourier transform of the wave vector of the optical field of the fluorescence in the environment.
Further, U in the step 4aL1The (ρ) acquisition method specifically includes:
step 41a, setting the point light source as an ideal spherical wave;
step 42a, calculating a scattering potential f (r) by using a formula (3) according to the three-dimensional refractive index distribution n (r);
Figure BDA0003015321130000036
Figure BDA0003015321130000037
in the formula, nmIs the average refractive index, k, of the environment in which the sample is locatedmWave vector of light field of illumination laser in environment in SIM mode, lambda0A wavelength of an illumination laser in the SIM mode;
step 43a, according to f (r), calculating UL1(ρ)。
Further, U in the step 43aL1(ρ) is solved by using a Lippmann-schwigger model in an integral form, or using a multilayer beam propagation model provided by the following equations (5) to (7):
Ul(ρ,(l+1)Δz)＝Ft-1{Pprop(ξ,Δz)·Ft{Ul(ρ,lΔz)}} (5)
Figure BDA0003015321130000041
Figure BDA0003015321130000042
Figure BDA0003015321130000043
Figure BDA0003015321130000044
in the formula of Ul(ρ, (l +1) Δ z) is the light field distribution, U, of the point light source propagating directly from the imaging focal plane corresponding to the original fluorescence sub-image to the ith layer of the sample without sample actionl(p, l Δ z) is the total light field distribution of the point light source propagating from the corresponding imaging focal plane of the original fluorescence sub-image to layer l-1 of the sample,
Figure BDA0003015321130000045
is Ul(ρ, (l) Δ z) the scattered light field, U, generated by the interaction with the sample after passing through the l-th layer of said samplel+1(ρ, (L +1) Δ z) is the total light field distribution of the point light source propagating from the imaging focal plane corresponding to the original fluorescence sub-image to the L-th layer of the sample, L is the number of layers of the point light source from the imaging focal plane corresponding to the original fluorescence sub-image to the outer surface of the sample, L is 1 … …, L, z is the coordinate distribution of the optical axis of the SIM-mode, Δ z is the thickness of each layer of the sample, Ft {. cndot.) represents the fourier transform, Ft {. cndot-1{. represents an inverse Fourier transform, Pprop(xi, Δ z) is an angular spectrum propagation function represented by formula (8), G (xi, Δ z) is a green function represented by formula (9), fl(p) is the scattering potential of the point light source propagating from the imaging focal plane corresponding to the original fluorescent sub-image to the ith layer of the sample, when l is 1, U is1(rho, Delta z) is taken as USPH(ρ,Δz)。
The invention also provides a self-adaptive correction method of the super-resolution microscope image, which comprises the following steps:
the method for adaptively correcting a SIM super-resolution image from an illumination path using a three-dimensional refractive index distribution obtained through an original phase image of an ODT modality of a sample includes:
step 1b, estimating and solving illumination parameters of each illumination mode according to the original fluorescence image of the SIM mode of the sample;
step 2b, generating uniform ideal sinusoidal illumination mode distribution at the outer surface of the sample according to the prior information of the illumination mode in sinusoidal distribution and the illumination parameters obtained by the first module;
step 3b, calculating the total number L of layers from the imaging focal plane corresponding to the original fluorescence image to the outer surface of the sample along the optical axis z axis of the SIM mode and the second light field distribution U of the ideal sine distribution light source transmitted from the outer surface of the sample to the imaging focal plane of the original fluorescence image according to the three-dimensional refractive index distributionL2(ρ) where ρ is (x, y) a two-dimensional spatial coordinate perpendicular to the z-axis;
step 4b, according to UL2(ρ) obtaining the illumination pattern after scattering distortion;
and 5b, iterating the illumination mode after the scattering distortion by using the optimization model represented by the formula (10), and reconstructing to obtain the optimal SIM super-resolution image
Figure BDA0003015321130000051
Figure BDA0003015321130000052
Where num is the total number of frames of the original fluorescence image in each period of the illumination pattern, DiFor the i frame of the original fluorescence image, H is a second optical transfer function H2(xi) point spread function, g, obtained after inverse Fourier transformiComprises the following steps: gi(ρ)＝I′i(ρ)·o(ρ)，I′i(ρ) and I'iIs DiCorresponding to said scatter-distorted illumination pattern, hess (o) representing a hessian matrix, | · |1And L1 norm representing the matrix, o and o (rho) are SIM super-resolution images to be reconstructed, lambda is a data fidelity term coefficient, alpha is a Hessian continuity constraint coefficient, and beta is a sparse term constraint coefficient.
Further, the illumination mode after the scattering distortion in the step 4b is represented as I' (ρ), which is obtained by equation (12):
I′(ρ)＝|UL2(ρ)|2 (12)。
further, U in the step 3bL2The (ρ) acquisition method specifically includes:
step 31b, calculating a scattering potential f (r) by using the formula (3) according to the three-dimensional refractive index distribution n (r);
Figure BDA0003015321130000053
Figure BDA0003015321130000054
in the formula, nmIs the average refractive index, k, of the environment in which the sample is locatedmWave vector of light field of illumination laser in environment in SIM mode, lambda0A wavelength of an illumination laser in the SIM mode;
step 32b, according to f (r), calculating UL2(ρ)。
Further, U in the step 32bL2(ρ) is solved by using a Lippmann-schwigger model in an integral form or using a multilayer beam propagation model provided by the following equations (5) to (7):
Ul(ρ,(l+1)Δz)＝Ft-1{Pprop(ξ,Δz)·Ft{Ul(ρ,lΔz)}} (5)
Figure BDA0003015321130000055
Figure BDA0003015321130000056
Figure BDA0003015321130000057
Figure BDA0003015321130000061
in the formula of Ul(ρ, (l +1) Δ z) is the light field distribution, U, for the ideal sinusoidal illumination mode distribution propagating directly from the outer surface of the sample to the l-th layer of the sample without sample effectl(p, l Δ z) is the total light field distribution for the ideal sinusoidal illumination mode distribution to propagate from the outer surface of the sample to the l-1 st layer of the sample,
Figure BDA0003015321130000062
is Ul(ρ, (l) Δ z) the scattered light field, U, generated by the interaction with the sample after passing through the l-th layer of said samplel+1(ρ, (L +1) Δ z) is the total light field distribution of the ideal sinusoidal illumination mode distribution propagating from the outer surface of the sample to the L-th layer of the sample, L is the number of layers of the imaging focal plane corresponding to the ideal sinusoidal illumination mode distribution from the original fluorescence image of the SIM modality of the sample, L-1 … …, L, z is the coordinate distribution of the optical axis of the SIM modality, Δ z is the thickness of each layer of the sample, Ft {. cndot } represents the fourier transform, Ft {. cndot-1{. represents an inverse Fourier transform, Pprop(xi, Δ z) is an angular spectrum propagation function represented by formula (8), G (xi, Δ z) is a green function represented by formula (9), fl(p) is the scattering potential of the ideal sinusoidal illumination mode distribution propagating from the imaging focal plane corresponding to the original fluorescence image of the SIM modality of the sample to the ith layer of the sample, when l is 1, U1(ρ, Δ z) takes the ideal sinusoidal illumination pattern generated in step 2 b.
Further, H in said step 5b2(xi) is obtained by one of two ways:
the method comprises the steps of firstly, constructing an optical transfer function by using an ideal model to obtain;
and secondly, acquiring a point spread function H (rho) by fluorescent bead shooting, and acquiring H by using the H (rho)2(ξ)，H2(xi) ═ Ft { h (ρ) }, Ft is FourierAnd (5) carrying out a Rie transform.
Further, H in said step 5b2(ξ) is obtained by:
step 51b, using a three-dimensional refractive index profile obtained by an original phase image of an ODT mode of a sample;
step 52b, reconstructing to obtain an initial SIM super-resolution image with an artifact according to the optical transfer function and the original fluorescence image of the sample in the SIM mode;
step 53b, dividing the original fluorescence image into a plurality of original fluorescence sub-images according to the first division mode, wherein the edges of the adjacent sub-images are overlapped with each other;
step 54b, finding out an imaging focal plane corresponding to each original fluorescence sub-image containing the fluorescence signal and the number of sample layers corresponding to the imaging focal plane along the optical axis z axis of the SIM mode through the three-dimensional refractive index distribution, and arranging a point light source at the center of each original fluorescence sub-image containing the fluorescence signal and a selected point of an adjacent region of the center;
step 55b, calculating a first light field distribution U of the point light source transmitted to the surface of the sample from the imaging focal plane corresponding to the original fluorescent sub-image according to the three-dimensional refractive index distributionL1(ρ), ρ ═ (x, y) is the two-dimensional spatial coordinate perpendicular to z, and L is the total number of layers of the sample from the imaging focal plane corresponding to the original fluorescence sub-image to the outer surface of the sample;
step 56b, according to UL1(p), calculating a first coherent transfer function of the imaging focal plane corresponding to the original fluorescence sub-image
Figure BDA0003015321130000071
And is composed of
Figure BDA0003015321130000072
Updating the second optical transfer function H2(ξ)，
Figure BDA0003015321130000073
Figure BDA0003015321130000074
ξ is a two-dimensional coordinate in the frequency domain, which represents an autocorrelation calculation.
The invention also provides a SIM-ODT bimodal system, which comprises: an ODT lighting module for generating ODT mode laser light; the SIM super-resolution laser lighting module is used for generating SIM modal laser in sinusoidal distribution; the first objective lens is arranged between one side of the sample and the light outlet of the ODT lighting module and is used for irradiating the sample by the laser generated by the ODT lighting module; the second objective is arranged on the other side of the sample and is used for receiving ODT (optical emission tomography) mode laser after passing through the sample, SIM (subscriber identity module) mode laser passing through and irradiating the sample and SIM mode original fluorescence generated by the sample; the SIM super-resolution image acquisition module is used for acquiring an original fluorescence image of the SIM modality; the ODT image acquisition module is used for acquiring an original phase image of an ODT modality; the sample three-dimensional refractive index distribution calculation module is used for processing the original phase image by utilizing a reconstruction algorithm of diffraction tomography, and reconstructing to obtain the three-dimensional refractive index distribution of the unmarked sample; an adaptive image correction module for adaptively correcting the optical transfer function and/or distorted illumination pattern of the SIM modality using the adaptive correction method of the super resolution microscope image as described above; the 2D/3D-SIM super-resolution image reconstruction module is used for reconstructing an image of the original fluorescence image and the optical transfer function according to the optical transfer function or/and the distorted illumination mode corrected by the self-adaptive image correction module and the original fluorescence image to obtain a self-adaptive corrected SIM super-resolution image; and the bimodal image fusion module is used for fusing the SIM super-resolution image subjected to self-adaptive correction with the three-dimensional refractive index distribution according to a spatial position relationship, and simultaneously carrying out three-dimensional high-resolution dynamic imaging and deep specific two-dimensional or three-dimensional super-resolution dynamic imaging on the sample.
The invention obtains the three-dimensional refractive index information of the sample by utilizing the optical diffraction tomography mode in the dual-mode microscope system, and then adaptively corrects the two-dimensional or three-dimensional structure optical super-resolution image of the corresponding region from the illumination path and the detection path by utilizing the local refractive index information.
Drawings
FIG. 1 is a schematic diagram of a SIM-ODT bimodal system according to an embodiment of the present invention.
Fig. 2 is a block flow diagram of an adaptive calibration detection path according to an embodiment of the present invention.
Fig. 3 is a block flow diagram of an adaptive illumination path correction method according to an embodiment of the present invention.
Fig. 4 is a block diagram of a process for adaptively correcting a detection path and an illumination path according to an embodiment of the present invention.
Detailed Description
The invention is described in detail below with reference to the figures and examples.
As shown in fig. 1, an SIM-ODT bimodal system provided in an embodiment of the present invention includes an ODT illumination module, an SIM super-resolution laser illumination module, a first objective lens, a second objective lens, an SIM super-resolution image acquisition module, an ODT image acquisition module, a sample three-dimensional refractive index distribution calculation module, an adaptive image correction module, a 2D/3D-SIM super-resolution image reconstruction module, and a bimodal image fusion module, where:
the ODT lighting module is used for generating ODT modal laser and carrying out multi-angle unmarked diffraction chromatography lighting on the sample, and the used laser is required to have high coherence. Wherein the sample is typically a single cell thick, in the range of 10 μm to 20 μm.
The SIM super-resolution laser lighting module is used for generating SIM modal laser in sinusoidal distribution.
The first objective lens is arranged between one side of the sample and the light outlet of the ODT lighting module and is used for enabling the laser generated by the ODT lighting module to irradiate the sample.
The second objective lens is arranged on the other side of the sample and is used for receiving ODT mode laser after passing through the sample, SIM mode laser passing through and irradiating the sample and SIM mode original fluorescence generated by the sample. In order to realize SIM super-resolution imaging, the second objective lens is a high-power objective lens, and the numerical aperture of the second objective lens is larger than or equal to that of the first objective lens, so that the laser of the ODT mode passing through the first objective lens can be received by the second objective lens.
And the SIM super-resolution image acquisition module is used for acquiring the original fluorescence image of the SIM modality.
The ODT image acquisition module is used for acquiring an original phase image of an ODT modality.
And the sample three-dimensional refractive index distribution calculation module is used for processing the original phase image by utilizing a reconstruction algorithm of diffraction tomography, such as Born approximate reconstruction, Rytov approximate reconstruction, multilayer propagation approximate reconstruction or other reconstruction methods, and reconstructing to obtain the unmarked three-dimensional refractive index distribution of the sample.
The adaptive image correction module is used for carrying out adaptive correction on the optical transfer function and/or the distorted illumination mode of the SIM modality by utilizing the imaging detection path and/or the illumination path. The imaging detection path refers to a process of emitting fluorescence from the sample, wherein the fluorescence propagates in the sample and is received by the second objective lens and then is acquired by the SIM super-resolution image acquisition module. The illumination path refers to a process that the SIM super-resolution laser illumination module emits laser, and the laser is irradiated to a sample imaging focal plane through the second objective lens to form a sine-distributed illumination mode.
And the 2D/3D-SIM super-resolution image reconstruction module carries out image reconstruction on the original fluorescence image and the optical transfer function according to the optical transfer function or/and the distorted illumination mode corrected by the self-adaptive image correction module and the original fluorescence image to obtain a self-adaptively corrected SIM super-resolution image.
The image reconstruction method can be a traditional wiener deconvolution method, high-frequency components and low-frequency components in an image formed by the original fluorescence are decoupled, the high-frequency components are moved to positions corresponding to the illumination frequency, and finally the low-frequency components and the high-frequency components are integrated to obtain a super-resolution image; or the illumination mode is iterated integrally, and an optimal super-resolution image is found by utilizing an optimization method.
The bimodal image fusion module is used for fusing the SIM super-resolution image after self-adaptive correction and the three-dimensional refractive index distribution according to a spatial position relation, and simultaneously carrying out three-dimensional high-resolution dynamic imaging and deep specific two-dimensional or three-dimensional super-resolution dynamic imaging on the sample. Where "fusion" may be understood as interpolating the size of individual pixels of the ODT modality and the SIM modality to the same value, and then superimposing the results of the two modalities to show the effect of different channel colors in one image.
As shown in fig. 1 and fig. 2, an adaptive correction method for a super-resolution microscope image according to an embodiment of the present invention includes:
the method for adaptively correcting the SIM super-resolution image from the imaging detection path by using the three-dimensional refractive index distribution obtained through the original phase image of the ODT mode of the sample specifically comprises the following steps:
step 1a, reconstructing and obtaining an initial SIM super-resolution image with an artifact according to an optical transfer function and an original fluorescence image of the sample in the SIM mode. The optical transfer function can be constructed by using an ideal model or an optical transfer function obtained by shooting fluorescent beads. The reconstruction method can utilize a traditional wiener deconvolution image reconstruction method, namely: firstly, estimating illumination parameters such as illumination frequency, illumination phase and illumination modulation depth coefficient of each illumination direction by using spectral correlation; then, decoupling the high-frequency component and the low-frequency component in the original fluorescence image, and moving the high-frequency component to a position corresponding to the illumination frequency; and finally, integrating the low-frequency component and the high-frequency component to obtain an initial SIM super-resolution image with an artifact.
And 2a, dividing the initial SIM super-resolution image into a plurality of first SIM super-resolution sub-images according to a first dividing mode, dividing the original fluorescent image into a plurality of original fluorescent sub-images according to the first dividing mode, and overlapping the edges of the adjacent sub-images. The first division manner may be a uniform division manner, and may also be understood as: the divided image is divided into a plurality of pixels in the horizontal and vertical directions according to a grid division mode, wherein the number of the pixels in the horizontal and vertical directions is equal to or not less than 64 pixels. The number of pixels of the part where the edges of the adjacent sub-images overlap with each other is not less than 10% of the number of pixels of a single sub-image, so that the continuity of the subsequent image splicing can be ensured.
And 3a, finding out an imaging focal plane corresponding to each original fluorescence sub-image containing the fluorescence signal and the number of sample layers corresponding to the imaging focal plane along the optical axis z axis of the SIM mode through the three-dimensional refractive index distribution, and arranging a point light source at the center of each original fluorescence sub-image containing the fluorescence signal and a selected point of an adjacent region of the center. The selected point of the neighboring area in the center may be set as an eight-neighborhood diagonal point of the center point of the image, but is not limited thereto, and a person skilled in the art may select and set the selected point according to actual requirements. For convenience of calculation, the point light source can be replaced by an ideal spherical wave, as shown in formula (1):
Figure BDA0003015321130000101
wherein, USPH(r) is the optical field distribution of an ideal spherical wave, A is the spherical wave amplitude, i is the imaginary unit, k is the wave vector in free space, phi0For the initial phase, r ═ (x, y, z) is the three-dimensional spatial coordinate.
Step 4a, calculating a first light field distribution U of the point light source transmitted to the surface of the sample from the imaging focal plane corresponding to the original fluorescent subimage according to the three-dimensional refractive index distributionL1(ρ) where ρ is (x, y) is the two-dimensional spatial coordinate perpendicular to z, and L is the total number of layers of the sample from the imaging focal plane corresponding to the original fluorescence sub-image to the outer surface of the sample.
Step 5a, according to UL1(p), calculating a first coherent transfer function of the imaging focal plane corresponding to the original fluorescence sub-image
Figure BDA0003015321130000102
And is composed of
Figure BDA0003015321130000103
Updating the first optical transfer function H1(ξ), ξ are two-dimensional coordinates in the frequency domain. Preferably, the first and second electrodes are formed of a metal,
Figure BDA0003015321130000104
: |, represents an auto-correlation calculation. Of course, H1(ξ) may also be obtained in other ways known in the art.
In step 5a
Figure BDA0003015321130000105
Can be obtained by the first method, the second method and other existing methods.
Method one, according to the focal length of the second objective lens, the U is adjustedL1(rho) is positively propagated to the pupil surface position by utilizing a Fresnel diffraction formula to obtain
Figure BDA0003015321130000106
The second lens is used for receiving ODT mode laser after passing through the sample, laser generated by the SIM super-resolution laser lighting module irradiates the sample after passing through the sample, and original fluorescence of the SIM mode generated by the sample.
Secondly, calculating U by utilizing refocusing back propagation provided by the formula (1)L1(ρ) obtaining
Figure BDA0003015321130000107
Figure BDA0003015321130000108
Figure BDA0003015321130000109
In the formula of UPSF(p) is UL1(ρ) light field distribution propagating back to the imaging focal plane in the opposite direction, i.e. the coherence point spread function, ρLIs the coordinate of any place on the outer surface of the sample, zLFor the distance of the imaging focal plane from the outer surface of the sample, P (-) characterizes the pupil function, UL(ρL) Is the light field distribution of the outer surface of the sample, j is the unit of imaginary number, kmLight field in-loop for illumination laser of SIM modeThe wave vector in the environment, Ft {. is Fourier transform.
Further, to
Figure BDA00030153211300001010
The phase of the signal is subjected to phase unwrapping, and decomposition and reconstruction are carried out by utilizing a Zernike polynomial, so that noise interference is removed. The module can perform arithmetic averaging on coherent transfer functions obtained by different point light sources in an area corresponding to the same original fluorescent subimage, can also perform clustering on Zernike polynomial decomposition results obtained by the point light sources by using a clustering method, or can obtain an average coherent transfer function by using other methods.
In one embodiment, U in said step 4aL1The (ρ) acquisition method specifically includes:
step 41a, setting the point light source as an ideal spherical wave.
Step 42a, calculating a scattering potential f (r) by using a formula (3) according to the three-dimensional refractive index distribution n (r);
Figure BDA0003015321130000111
Figure BDA0003015321130000112
in the formula, nmThe average refractive index of the environment (such as culture solution) where the sample is located can be obtained by an Abbe refractometer, and the refractive index of the cell culture solution used in actual calculation is close to the refractive index of water of 1.333 and is an empirical value; k is a radical ofmWave vector of light field of illumination laser in environment in SIM mode, lambda0The wavelength of the illumination laser in the SIM mode.
Step 43a, according to f (r), calculating UL1(ρ)。
In one embodiment, U in said step 43aL1(p) is obtained by using an integral-form Lippmann-Schwigger model, and can also be obtained by multilayer beam propagation as provided in the following equations (5) to (7)The broadcast model is solved, and can be obtained by other existing formulas:
Ul(ρ,(l+1)Δz)＝Ft-1{Pprop(ξ,Δz)·Ft{Ul(ρ,lΔz)}} (5)
Figure BDA0003015321130000113
Figure BDA0003015321130000114
Figure BDA0003015321130000115
Figure BDA0003015321130000116
in the formula of Ul(ρ, (l +1) Δ z) is the light field distribution, U, for the ideal sinusoidal illumination mode distribution propagating directly from the outer surface of the sample to the l-th layer of the sample without sample effectl(p, l Δ z) is the total light field distribution for the ideal sinusoidal illumination mode distribution to propagate from the outer surface of the sample to the l-1 st layer of the sample,
Figure BDA0003015321130000117
is Ul(ρ, (l) Δ z) the scattered light field, U, generated by the interaction with the sample after passing through the l-th layer of said samplel+1(ρ, (L +1) Δ z) is the total light field distribution of the ideal sinusoidal illumination mode distribution propagating from the outer surface of the sample to the L-th layer of the sample, L1 … …, L, z is the coordinate distribution of the optical axis of the SIM mode, Δ z is the thickness of each layer of the sample, Ft {. cndot } represents the fourier transform, Ft {. cndot-1{. represents an inverse Fourier transform, Pprop(xi, Δ z) is an angular spectrum propagation function represented by formula (8), G (xi, Δ z) is a green function represented by formula (9), fl(p) is the corresponding composition of the point light source from the original fluorescence sub-imageThe image focal plane propagates to the scattering potential of the sample ith layer.
The multilayer beam propagation model is a layer-by-layer propagation module, n takes the total layer number L from 1 to the sample from the imaging focal plane corresponding to the original fluorescence sub-image to the outer surface of the sample, and when L is 1, U is equal to 11(rho, Delta z) is taken as USPH(ρ,Δz)。
Step 6a, according to H1And (ξ) and the corresponding original fluorescence sub-image, and reconstructing again to obtain a second SIM super-resolution sub-image of the original fluorescence sub-image, wherein the artifacts are reduced. The reconstruction method may be a conventional wiener deconvolution method, or may be an iterative optimization method.
And 7a, splicing the second SIM super-resolution sub-image reconstructed in the step 6a and the first SIM super-resolution sub-image corresponding to the region where the original fluorescence sub-image without the fluorescence signal is located into a final complete SIM super-resolution image. In the splicing process, the edge fusion of adjacent sub-images can use a weighted fusion method or other fusion methods to ensure that the overlapped part is smoothly transited from the region corresponding to the previous sub-image to the region corresponding to the next sub-image, thereby ensuring the continuity of the images. For each region without fluorescence signal distribution in the SIM super-resolution image, the corresponding first SIM super-resolution sub-image is directly used, so that the calculation amount can be reduced.
Iterative optimization can also be performed on the final complete SIM super-resolution image output in the step 7a, so that the noise of the super-resolution image is reduced, and the image resolution is increased. The iteration optimization model can add a Hessian regular term and a sparse regular term to jointly modulate the optimization process.
As shown in fig. 1 and 3, an adaptive correction method for a super-resolution microscope image according to an embodiment of the present invention includes:
the method for adaptively correcting a SIM super-resolution image from an illumination path using a three-dimensional refractive index distribution obtained through an original phase image of an ODT modality of a sample includes:
and step 1b, estimating the illumination parameters of each illumination mode according to the original fluorescence image of the SIM mode of the sample. Specifically, the original fluorescence image is multiplied by a phase difference matrix of an illumination mode to carry out phase difference separation, and high-frequency and low-frequency components of the SIM super-resolution image are obtained; then, the correlation between the obtained high-frequency and low-frequency components is calculated, and the correlation of the frequency spectrum is used to estimate the illumination parameters such as the illumination frequency, the illumination phase, the illumination modulation depth coefficient and the like of each illumination direction, wherein the phase difference matrix of the illumination mode is obtained according to the illumination mode used in the microscope hardware design.
And 2b, generating uniform ideal sinusoidal illumination mode distribution at the outer surface of the sample according to the prior information of the illumination mode in sinusoidal distribution and the illumination parameters obtained by the first module.
Step 3b, calculating the total number L of layers from the imaging focal plane corresponding to the original fluorescence image to the outer surface of the sample along the optical axis z axis of the SIM mode and a second light field distribution U of an ideal sine distribution light source transmitted from the outer surface of the sample to the imaging focal plane of the original fluorescence image according to the three-dimensional refractive index distributionL2(ρ) and ρ (x, y) are two-dimensional spatial coordinates perpendicular to the z-axis.
Wherein U in step 3bL2The (ρ) acquisition method specifically includes:
and 31b, calculating the scattering potential f (r) by using the formula (3) according to the three-dimensional refractive index distribution n (r).
Step 32b, according to f (r), calculating UL2(ρ)。UL2The (p) can be obtained by solving using an integral version of the Lippmann-schwigger model, the same multilayer beam propagation models (equations (5) to (7)) as in the above-described embodiment, and other existing methods.
Ul(ρ,(l+1)Δz)＝Ft-1{Pprop(ξ,Δz)·Ft{Ul(ρ,lΔz)}} (5)
Figure BDA0003015321130000131
Figure BDA0003015321130000132
Figure BDA0003015321130000133
Figure BDA0003015321130000134
In the formula of Ul(ρ, (l +1) Δ z) is the light field distribution, U, for the ideal sinusoidal illumination mode distribution propagating directly from the outer surface of the sample to the l-th layer of the sample without sample effectl(p, l Δ z) is the total light field distribution for the ideal sinusoidal illumination mode distribution to propagate from the outer surface of the sample to the l-1 st layer of the sample,
Figure BDA0003015321130000135
is Ul(ρ, (l) Δ z) the scattered light field, U, generated by the interaction with the sample after passing through the l-th layer of said samplel+1(ρ, (L +1) Δ z) is the total light field distribution of the ideal sinusoidal illumination mode distribution propagating from the outer surface of the sample to the L-th layer of the sample, L is the number of layers of the imaging focal plane corresponding to the ideal sinusoidal illumination mode distribution from the original fluorescence image of the SIM modality of the sample, L-1 … …, L, z is the coordinate distribution of the optical axis of the SIM modality, Δ z is the thickness of each layer of the sample, Ft {. cndot } represents the fourier transform, Ft {. cndot-1{. represents an inverse Fourier transform, Pprop(xi, Δ z) is an angular spectrum propagation function represented by formula (8), G (xi, Δ z) is a green function represented by formula (9), fl(p) is the scattering potential of the ideal sinusoidal illumination mode distribution propagating from the imaging focal plane corresponding to the original fluorescence image of the SIM modality of the sample to the ith layer of the sample, when l is 1, U1(ρ, Δ z) takes the ideal sinusoidal illumination pattern generated in step 2 b.
Step 4b, according to UL2(ρ) obtaining an illumination pattern I' (ρ) after scattering distortion due to unevenness of refractive index distribution of the thick sample, which is obtained by equation (11):
I′(ρ)＝|UL2(ρ)|2 (11)。
and 5b, iterating the illumination mode after the scattering distortion by using the optimization model represented by the formula (10), and reconstructing to obtain the optimal SIM super-resolution image
Figure BDA0003015321130000141
The artifacts of (a) are further reduced.
Figure BDA0003015321130000142
Where num is the total number of frames of the original fluorescence image in each period of the illumination pattern, DiFor the i frame of the original fluorescence image, H is a second optical transfer function H2(xi) point spread function, g, obtained after inverse Fourier transformiComprises the following steps: gi(ρ)＝I′i(ρ)·o(ρ)，I′i(ρ) and I'iIs DiCorresponding to said scatter-distorted illumination pattern, hess (o) representing a hessian matrix, | · |1The L1 norm of the matrix is represented, o and o (rho) are SIM super-resolution images to be reconstructed, lambda is a data fidelity term coefficient, the value range of the data fidelity term coefficient is generally between 200 (low signal-to-noise ratio) and 300 (high signal-to-noise ratio), beta is a sparse term constraint coefficient, the value of the sparse term constraint coefficient is generally a multiple relation of 1/20-1/5 of lambda, alpha is a Hessian continuity constraint coefficient, and the value setting principle is as follows: if the data are acquired at intervals, a small value is taken, such as 0.1 or 0.01, if the data are acquired at continuous time, a large value is taken, such as 5, 10 and 20, and if the processed image is too smooth, the value of alpha is reduced.
In the above embodiment, step 5b may also use an optimization model represented by formula (11) instead of formula (10):
Figure BDA0003015321130000143
when the illumination path is adopted alone for the adaptive correction, H in the step 5b2(xi) can be obtained by constructing an optical transfer function using an ideal model, or byObtaining a point spread function H (rho) by shooting through fluorescent beads, and obtaining H by using the H (rho)2(ξ)，H2(ξ) ═ Ft { h (ρ) }, Ft is the fourier transform.
As shown in fig. 4, the embodiment of the present invention provides an adaptive correction method combining the detection path correction method shown in fig. 2 and the illumination path correction method shown in fig. 3, and corrects both the detection path and the illumination path. The main idea of this combined approach is to simultaneously substitute both the first optical transfer function obtained in the detection path correction method shown in fig. 2 and the distorted illumination pattern obtained in the illumination path correction method shown in fig. 3 into the optimization model represented by equation (10).
Specifically, the adaptive method shown in fig. 4 specifically includes:
and step 1b, estimating the illumination parameters of each illumination mode according to the original fluorescence image of the SIM mode of the sample. Specifically, the original fluorescence image is multiplied by a phase difference matrix of an illumination mode to carry out phase difference separation, and high-frequency and low-frequency components of the SIM super-resolution image are obtained; then, the correlation between the obtained high-frequency and low-frequency components is calculated, and the correlation of the frequency spectrum is used to estimate the illumination parameters such as the illumination frequency, the illumination phase, the illumination modulation depth coefficient and the like of each illumination direction, wherein the phase difference matrix of the illumination mode is obtained according to the illumination mode used in the microscope hardware design.
And 2b, generating uniform ideal sinusoidal illumination mode distribution at the outer surface of the sample according to the prior information of the illumination mode in sinusoidal distribution and the illumination parameters obtained by the first module.
Step 3b, calculating the total number L of layers from the imaging focal plane corresponding to the original fluorescence image to the outer surface of the sample along the optical axis z axis of the SIM mode and a second light field distribution U of an ideal sine distribution light source transmitted from the outer surface of the sample to the imaging focal plane of the original fluorescence image according to the three-dimensional refractive index distributionL2(ρ) and ρ are two-dimensional spatial coordinates perpendicular to (x, y).
Wherein U in step 3bL2The (ρ) acquisition method specifically includes:
and 31b, calculating the scattering potential f (r) by using the formula (3) according to the three-dimensional refractive index distribution n (r).
Step 32b, according to f (r), calculating UL2(ρ)。UL2The (p) can be obtained by solving using an integral version of the Lippmann-schwigger model, using a multilayer beam propagation model provided by the following equations (5) to (7), and using other existing methods.
Ul(ρ,(l+1)Δz)＝Ft-1{Pprop(ξ,Δz)·Ft{Ul(ρ,lΔz)}} (5)
Figure BDA0003015321130000151
Figure BDA0003015321130000152
Figure BDA0003015321130000153
Figure BDA0003015321130000154
In the formula of Ul(ρ, (l +1) Δ z) is the light field distribution, U, for the ideal sinusoidal illumination mode distribution propagating directly from the outer surface of the sample to the l-th layer of the sample without sample effectl(p, l Δ z) is the total light field distribution for the ideal sinusoidal illumination mode distribution to propagate from the outer surface of the sample to the l-1 st layer of the sample,
Figure BDA0003015321130000155
is Ul(ρ, (l) Δ z) the scattered light field, U, generated by the interaction with the sample after passing through the l-th layer of said samplel+1(ρ, (l +1) Δ z) is the total light field distribution for the ideal sinusoidal illumination mode distribution propagating from the outer surface of the sample to the l-th layer of the sample, l is the SIM mode of the sample for the ideal sinusoidal illumination mode distributionThe number of imaging focal planes corresponding to the original fluorescence image of the state, L is 1 … …, L, z is the coordinate distribution of the optical axis of the SIM mode, Δ z is the thickness of each layer of the sample, Ft {. cndot } represents the Fourier transform, Ft {. cndot-1{. represents an inverse Fourier transform, Pprop(xi, Δ z) is an angular spectrum propagation function represented by formula (8), G (xi, Δ z) is a green function represented by formula (9), fl(p) is the scattering potential of the ideal sinusoidal illumination mode distribution propagating from the imaging focal plane corresponding to the original fluorescence image of the SIM modality of the sample to the ith layer of the sample, when l is 1, U1(ρ, Δ z) takes the ideal sinusoidal illumination pattern generated in step 2 b.
Step 4b, according to UL2(ρ) obtaining an illumination pattern I' (ρ) after scattering distortion due to unevenness of refractive index distribution of the thick sample, which is obtained by equation (12):
I′(ρ)＝|UL2(ρ)|2 (12)。
in formula (12), I'i(ρ) and I'iIs DiThe corresponding scattered distorted illumination pattern, and I' (ρ) here refers to the unspecified illumination pattern.
And 5b, iterating the illumination mode after the scattering distortion by using the optimization model represented by the formula (10), and reconstructing to obtain the optimal SIM super-resolution image
Figure BDA0003015321130000162
The artifacts of (a) are further reduced.
Figure BDA0003015321130000161
Where num is the total number of frames of the original fluorescence image in each period of the illumination pattern, DiFor the i frame of the original fluorescence image, H is a second optical transfer function H2(xi) point spread function, g, obtained after inverse Fourier transformiComprises the following steps: gi(ρ)＝I′i(ρ)·o(ρ)，I′i(ρ) and I'iIs DiCorresponding to said scattered distorted illumination pattern, Hess (o) denotes Hessian matrix, | |)1The L1 norm of the matrix is represented, o and o (rho) are SIM super-resolution images to be reconstructed, lambda is a data fidelity term coefficient, the value range of the data fidelity term coefficient is generally between 200 (low signal-to-noise ratio) and 300 (high signal-to-noise ratio), beta is a sparse term constraint coefficient, the value of the sparse term constraint coefficient is generally a multiple relation of 1/20-1/5 of lambda, alpha is a Hessian continuity constraint coefficient, and the value setting principle is as follows: if the data are acquired at intervals, a small value is taken, such as 0.1 or 0.01, if the data are acquired at continuous time, a large value is taken, such as 5, 10 and 20, and if the processed image is too smooth, the value of alpha is reduced.
H in said step 5b2(ξ) is obtained by:
step 51b, the three-dimensional refractive index profile obtained by the original phase image of the ODT modality of the sample is utilized.
And step 52b, reconstructing and obtaining an initial SIM super-resolution image with an artifact according to the optical transfer function and the original fluorescence image of the SIM mode of the sample.
And 53b, dividing the original fluorescent image into a plurality of original fluorescent sub-images according to the first division mode, wherein the edges of the adjacent sub-images are overlapped with each other.
And step 54b, finding out an imaging focal plane corresponding to each original fluorescence sub-image containing the fluorescence signal and the number of sample layers corresponding to the imaging focal plane along the optical axis z axis of the SIM mode through the three-dimensional refractive index distribution, and arranging a point light source at the center of each original fluorescence sub-image containing the fluorescence signal and the selected point of the adjacent region of the center.
Step 55b, calculating a first light field distribution U of the point light source transmitted to the surface of the sample from the imaging focal plane corresponding to the original fluorescent sub-image according to the three-dimensional refractive index distributionL1(ρ) where ρ is (x, y) is the two-dimensional spatial coordinate perpendicular to z, and L is the total number of layers of the sample from the imaging focal plane corresponding to the original fluorescence sub-image to the outer surface of the sample.
Step 56b, according to UL1(p), calculating a first coherent transfer function of the imaging focal plane corresponding to the original fluorescence sub-image
Figure BDA0003015321130000171
And is composed of
Figure BDA0003015321130000172
Updating the second optical transfer function H2(ξ)，
Figure BDA0003015321130000173
Figure BDA0003015321130000174
ξ is a two-dimensional coordinate in the frequency domain, which represents an autocorrelation calculation.
According to the invention, by utilizing ODT modal shooting in a dual-mode microscope system to obtain sample refractive index information and a wave optical propagation theory, the distortion of an illumination mode (attenuation pattern) at each position of an illumination path of the super-resolution microscope and an optical transfer function at each position of a detection path are corrected in a self-adaptive manner, and then the illumination mode and the optical transfer function information after the distortion correction are introduced into a reconstruction model, a more accurate image reconstruction method is designed.
Finally, it should be pointed out that: the above examples are only for illustrating the technical solutions of the present invention, and are not limited thereto. Those of ordinary skill in the art will understand that: modifications can be made to the technical solutions described in the foregoing embodiments, or some technical features may be equivalently replaced; such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims (10)

1. An adaptive correction method for super-resolution microscope images is characterized by comprising the following steps:
the method for self-adaptive correction of SIM super-resolution image from imaging detection path by using three-dimensional refractive index distribution obtained by original phase image of ODT mode of sample includes:
step 1a, reconstructing to obtain an initial SIM super-resolution image with an artifact according to an optical transfer function and an original fluorescence image of the sample in an SIM mode;
step 2a, dividing the initial SIM super-resolution image into a plurality of first SIM super-resolution sub-images according to a first dividing mode, dividing the original fluorescent image into a plurality of original fluorescent sub-images according to the first dividing mode, and overlapping the edges of the adjacent sub-images;
step 3a, finding out an imaging focal plane corresponding to each original fluorescence sub-image containing a fluorescence signal and a sample layer number corresponding to the imaging focal plane along an optical axis z axis of the SIM mode through the three-dimensional refractive index distribution, and arranging a point light source at the center of each original fluorescence sub-image containing the fluorescence signal and a selected point of an adjacent region of the center;
step 4a, calculating a first light field distribution U of the point light source transmitted to the surface of the sample from the imaging focal plane corresponding to the original fluorescent subimage according to the three-dimensional refractive index distributionL1(ρ), ρ ═ (x, y) is the two-dimensional spatial coordinate perpendicular to z, and L is the total number of layers of the sample from the imaging focal plane corresponding to the original fluorescence sub-image to the outer surface of the sample;
step 5a, according to UL1(p), calculating a first coherent transfer function of the imaging focal plane corresponding to the original fluorescence sub-image
Figure FDA0003015321120000011
And is composed of
Figure FDA0003015321120000012
Updating the first optical transfer function H1(ξ), ξ being a two-dimensional coordinate in the frequency domain;
step 6a, according to H1(ξ) and the corresponding original fluorescence sub-image, and reconstructing again to obtain a second SIM super-resolution sub-image of the original fluorescence sub-image;
and 7a, splicing the second SIM super-resolution sub-image reconstructed in the step 6a and the first SIM super-resolution sub-image corresponding to the region where the original fluorescence sub-image without the fluorescence signal is located into a final complete SIM super-resolution image.
2. The adaptive correction method for super resolution microscope images according to claim 1, wherein the step 5a is performed by
Figure FDA0003015321120000013
Obtained by one of the following two methods:
method one, according to the focal length of the second objective lens, the U is adjustedL1(rho) is positively propagated to the pupil surface position by utilizing a Fresnel diffraction formula to obtain
Figure FDA0003015321120000014
The second lens is used for receiving ODT modal laser after passing through the sample, irradiating the sample after the laser generated by the SIM super-resolution laser lighting module passes through the sample, and generating original fluorescence in an SIM mode generated by the sample; wherein the second lens is used for receiving ODT mode laser after passing through the sample, original fluorescence of SIM mode generated by the sample and allowing the SIM mode laser to pass through and irradiate the sample;
secondly, calculating U by utilizing refocusing back propagation provided by the formula (1)L1(ρ) obtaining
Figure FDA0003015321120000021
Figure FDA0003015321120000022
Figure FDA0003015321120000023
In the formula of UPSF(p) is UL1(ρ) light field propagating back to the imaging focal planeDistribution, i.e. coherent point spread function, pLIs the coordinate of any place on the outer surface of the sample, zLCharacterizing the pupil function, U, for the physical distance of the imaging focal plane to the outer surface of the sample, P (-), P (-. cndot.)L(ρL) Is the light field distribution of the outer surface of the sample, j is the unit of imaginary number, kmFt { } is the fourier transform, which is the wave vector of the optical field of fluorescence in the environment.
3. The adaptive correction method for super resolution microscope images according to claim 2, wherein U in step 4a is set to zeroL1The (ρ) acquisition method specifically includes:
step 41a, setting the point light source as an ideal spherical wave;
step 42a, calculating a scattering potential f (r) by using a formula (3) according to the three-dimensional refractive index distribution n (r);
Figure FDA0003015321120000024
Figure FDA0003015321120000025
in the formula, nmIs the average refractive index, k, of the environment in which the sample is locatedmWave vector of light field of illumination laser in environment in SIM mode, lambda0A wavelength of an illumination laser in the SIM mode;
step 43a, according to f (r), calculating UL1(ρ)。
4. The adaptive correction method for super resolution microscope images according to claim 3, wherein U in step 43a is set toL1(ρ) is solved by using a Lippmann-schwigger model in an integral form, or using a multilayer beam propagation model provided by the following equations (5) to (7):
Ul(ρ，(l+1)Δz)＝Ft-1{Pprop(ξ，Δz)·Ft{Ul(ρ，lΔz)}} (5)
Figure FDA0003015321120000031
Figure FDA0003015321120000032
Figure FDA0003015321120000033
Figure FDA0003015321120000034
in the formula of Ul(ρ, (l +1) Δ z) is the light field distribution, U, of the point light source propagating directly from the imaging focal plane corresponding to the original fluorescence sub-image to the ith layer of the sample without sample actionl(p, l Δ z) is the total light field distribution of the point light source propagating from the corresponding imaging focal plane of the original fluorescence sub-image to layer l-1 of the sample,
Figure FDA0003015321120000035
is Ul(ρ, (l) Δ z) the scattered light field, U, generated by the interaction with the sample after passing through the l-th layer of said samplel+1(ρ, (L +1) Δ z) is the total light field distribution of the point light source propagating from the imaging focal plane corresponding to the original fluorescence sub-image to the L-th layer of the sample, L is the number of layers of the point light source from the imaging focal plane corresponding to the original fluorescence sub-image to the outer surface of the sample, L1-1{. represents an inverse Fourier transform, Pprop(xi, Δ z) is an angular spectrum propagation function represented by formula (8), G (xi, Δ z) is a green function represented by formula (9), fl(p) is the point light source from the sourceThe scattering potential of the l layer of the sample is propagated by the imaging focal plane corresponding to the initial fluorescence sub-image, and when l is 1, U is1(rho, Delta z) is taken as USPH(ρ，Δz)。
5. An adaptive correction method for super-resolution microscope images is characterized by comprising the following steps:
the method for adaptively correcting a SIM super-resolution image from an illumination path using a three-dimensional refractive index distribution obtained through an original phase image of an ODT modality of a sample includes:
step 1b, estimating and solving illumination parameters of each illumination mode according to the original fluorescence image of the SIM mode of the sample;
step 2b, generating uniform ideal sinusoidal illumination mode distribution at the outer surface of the sample according to the prior information of the illumination mode in sinusoidal distribution and the illumination parameters obtained by the first module;
step 3b, calculating the total number L of layers from the imaging focal plane corresponding to the original fluorescence image to the outer surface of the sample along the optical axis z axis of the SIM mode and the second light field distribution U of the ideal sine distribution light source transmitted from the outer surface of the sample to the imaging focal plane of the original fluorescence image according to the three-dimensional refractive index distributionL2(ρ) where ρ is (x, y) a two-dimensional spatial coordinate perpendicular to the z-axis;
step 4b, according to UL2(ρ) obtaining the illumination pattern after scattering distortion;
and 5b, iterating the illumination mode after the scattering distortion by using the optimization model represented by the formula (10), and reconstructing to obtain the optimal SIM super-resolution image
Figure FDA0003015321120000041
Figure FDA0003015321120000042
Where num is the total number of frames of the original fluorescence image in each period of the illumination pattern, DiFor the ith frameOriginal fluorescence image, H is the second optical transfer function H2(xi) point spread function, g, obtained after inverse Fourier transformiComprises the following steps: gi(ρ)＝I′i(ρ)·o(ρ)，I′i(ρ) and I'iHess (o) represents the Hessian matrix, | · |. the luminance | |, for the illumination pattern corresponding to Di after the scattering distortion1And L1 norm representing the matrix, o and o (rho) are SIM super-resolution images to be reconstructed, lambda is a data fidelity term coefficient, alpha is a Hessian continuity constraint coefficient, and beta is a sparse term constraint coefficient.
6. The adaptive correction method for super resolution microscope images according to claim 5, wherein U in step 3bL2The (ρ) acquisition method specifically includes:
step 31b, calculating a scattering potential f (r) by using the formula (3) according to the three-dimensional refractive index distribution n (r);
Figure FDA0003015321120000043
Figure FDA0003015321120000044
in the formula, nmIs the average refractive index, k, of the environment in which the sample is locatedmWave vector of light field of illumination laser in environment in SIM mode, lambda0A wavelength of an illumination laser in the SIM mode;
step 32b, according to f (r), calculating UL2(ρ)。
7. The adaptive correction method for super resolution microscope images according to claim 6, wherein U in step 32bL2(ρ) is solved by using a Lippmann-schwigger model in an integral form or using a multilayer beam propagation model provided by the following equations (5) to (7):
Ul(ρ，(l+1)Δz)＝Ft-1{Pprop(ξ，Δz)·Ft{Ul(ρ，lΔz)}} (5)
Figure FDA0003015321120000045
Figure FDA0003015321120000046
Figure FDA0003015321120000051
Figure FDA0003015321120000052
in the formula of Ul(ρ, (l +1) Δ z) is the light field distribution, U, for the ideal sinusoidal illumination mode distribution propagating directly from the outer surface of the sample to the l-th layer of the sample without sample effectl(p, l Δ z) is the total light field distribution for the ideal sinusoidal illumination mode distribution to propagate from the outer surface of the sample to the l-1 st layer of the sample,
Figure FDA0003015321120000053
is Ul(ρ, (l) Δ z) the scattered light field, U, generated by the interaction with the sample after passing through the l-th layer of said samplel+1(ρ, (L +1) Δ z) is the total light field distribution of the ideal sinusoidal illumination mode distribution propagating from the outer surface of the sample to the L-th layer of the sample, L is the number of layers of the imaging focal plane corresponding to the ideal sinusoidal illumination mode distribution from the original fluorescence image of the SIM modality of the sample, L1-1{. represents an inverse Fourier transform, Pprop(xi, Δ z) is an angular spectrum propagation function represented by formula (8), G (xi, Δ z) is a green function represented by formula (9), fl(ρ) isThe ideal sinusoidal illumination mode distribution is transmitted to the scattering potential of the l-th layer of the sample from the imaging focal plane corresponding to the original fluorescence image of the SIM mode of the sample, and when l is 1, U is1(ρ, Δ z) takes the ideal sinusoidal illumination pattern generated in step 2 b.
8. The adaptive correction method for super resolution microscope images according to any one of claims 5 to 7, wherein H in step 5b2(xi) is obtained by one of two ways:
the method comprises the steps of firstly, constructing an optical transfer function by using an ideal model to obtain;
and secondly, acquiring a point spread function H (rho) by fluorescent bead shooting, and acquiring H by using the H (rho)2(ξ)，H2(ξ) ═ Ft { h (ρ) }, Ft is the fourier transform.
9. The adaptive correction method for super resolution microscope images according to claim 5, wherein H in step 5b2(ξ) is obtained by:
step 51b, using a three-dimensional refractive index profile obtained by an original phase image of an ODT mode of a sample;
step 52b, reconstructing to obtain an initial SIM super-resolution image with an artifact according to the optical transfer function and the original fluorescence image of the sample in the SIM mode;
step 53b, dividing the original fluorescence image into a plurality of original fluorescence sub-images according to the first division mode, wherein the edges of the adjacent sub-images are overlapped with each other;
step 54b, finding out an imaging focal plane corresponding to each original fluorescence sub-image containing the fluorescence signal and the number of sample layers corresponding to the imaging focal plane along the optical axis z axis of the SIM mode through the three-dimensional refractive index distribution, and arranging a point light source at the center of each original fluorescence sub-image containing the fluorescence signal and a selected point of an adjacent region of the center;
step 55b, calculating the corresponding imaging of the point light source from the original fluorescent subimage according to the three-dimensional refractive index distributionFirst light field distribution U of focal plane propagation to sample surfaceL1(ρ), ρ ═ (x, y) is the two-dimensional spatial coordinate perpendicular to z, and L is the total number of layers of the sample from the imaging focal plane corresponding to the original fluorescence sub-image to the outer surface of the sample;
step 56b, according to UL1(p), calculating a first coherent transfer function of the imaging focal plane corresponding to the original fluorescence sub-image
Figure FDA0003015321120000061
And is composed of
Figure FDA0003015321120000062
Updating the second optical transfer function H2(ξ)，
Figure FDA0003015321120000063
Figure FDA0003015321120000064
ξ is a two-dimensional coordinate in the frequency domain, which represents an autocorrelation calculation.
10. A SIM-ODT bimodal system, comprising:
an ODT lighting module for generating ODT mode laser light;
the SIM super-resolution laser lighting module is used for generating SIM modal laser in sinusoidal distribution;
the first objective lens is arranged between one side of the sample and the light outlet of the ODT lighting module and is used for irradiating the sample by the laser generated by the ODT lighting module;
the second objective is arranged on the other side of the sample and is used for receiving ODT (optical emission tomography) mode laser after passing through the sample, SIM (subscriber identity module) mode laser passing through and irradiating the sample and SIM mode original fluorescence generated by the sample;
the SIM super-resolution image acquisition module is used for acquiring an original fluorescence image of the SIM modality;
the ODT image acquisition module is used for acquiring an original phase image of an ODT modality;
the sample three-dimensional refractive index distribution calculation module is used for processing the original phase image by utilizing a reconstruction algorithm of diffraction tomography, and reconstructing to obtain the three-dimensional refractive index distribution of the unmarked sample;
an adaptive image correction module for adaptively correcting the optical transfer function and/or the distorted illumination pattern of the SIM modality with the adaptive correction method of super resolution microscope images as claimed in claim 1 or 5 or 9;
the 2D/3D-SIM super-resolution image reconstruction module is used for reconstructing an image of the original fluorescence image and the optical transfer function according to the optical transfer function or/and the distorted illumination mode corrected by the self-adaptive image correction module and the original fluorescence image to obtain a self-adaptive corrected SIM super-resolution image;
and the bimodal image fusion module is used for fusing the SIM super-resolution image subjected to self-adaptive correction with the three-dimensional refractive index distribution according to a spatial position relationship, and simultaneously carrying out three-dimensional high-resolution dynamic imaging and deep specific two-dimensional or three-dimensional super-resolution dynamic imaging on the sample.
CN202110386643.4A 2021-04-12 2021-04-12 Self-adaptive correction method of super-resolution microscope image and SIM-ODT (subscriber identity module-ODT) bimodal system Active CN113112405B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110386643.4A CN113112405B (en) 2021-04-12 2021-04-12 Self-adaptive correction method of super-resolution microscope image and SIM-ODT (subscriber identity module-ODT) bimodal system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110386643.4A CN113112405B (en) 2021-04-12 2021-04-12 Self-adaptive correction method of super-resolution microscope image and SIM-ODT (subscriber identity module-ODT) bimodal system

Publications (2)

Publication Number Publication Date
CN113112405A true CN113112405A (en) 2021-07-13
CN113112405B CN113112405B (en) 2022-04-12

Family

ID=76715406

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110386643.4A Active CN113112405B (en) 2021-04-12 2021-04-12 Self-adaptive correction method of super-resolution microscope image and SIM-ODT (subscriber identity module-ODT) bimodal system

Country Status (1)

Country Link
CN (1) CN113112405B (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115100033A (en) * 2022-05-20 2022-09-23 浙江大学 Fluorescence microscopic image super-resolution reconstruction method and device and computing equipment
WO2024065094A1 (en) * 2022-09-26 2024-04-04 广州超视计生物科技有限公司 Super-resolution microscopic imaging method and apparatus, and computer device and storage medium

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104515759A (en) * 2014-12-16 2015-04-15 中国科学院苏州生物医学工程技术研究所 Non-linear structure light illumination microscopic imaging method and system
CN105300941A (en) * 2015-10-20 2016-02-03 北京大学 Fluorescent optical sheet microscopic imaging system and method
CN109035143A (en) * 2018-07-17 2018-12-18 华中科技大学 A kind of three-dimensional super-resolution method based on the imaging of Bezier mating plate
US10169852B1 (en) * 2018-07-03 2019-01-01 Nanotronics Imaging, Inc. Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging
CN109712072A (en) * 2018-12-15 2019-05-03 浙江大学 Striped illumination Fourier iteration based on total internal reflection updates super-resolution micro imaging method
CN111077121A (en) * 2019-12-06 2020-04-28 中国科学院西安光学精密机械研究所 Rapid method and system for directly reconstructing structured light illumination super-resolution image in space domain
CN111127320A (en) * 2019-12-23 2020-05-08 哈尔滨工业大学（威海） Photoacoustic image super-resolution reconstruction method and device based on deep learning
CN111145089A (en) * 2019-11-18 2020-05-12 中国科学院苏州生物医学工程技术研究所 High fidelity image reconstruction method, system, computer equipment and storage medium

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104515759A (en) * 2014-12-16 2015-04-15 中国科学院苏州生物医学工程技术研究所 Non-linear structure light illumination microscopic imaging method and system
CN105300941A (en) * 2015-10-20 2016-02-03 北京大学 Fluorescent optical sheet microscopic imaging system and method
US10169852B1 (en) * 2018-07-03 2019-01-01 Nanotronics Imaging, Inc. Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging
CN109035143A (en) * 2018-07-17 2018-12-18 华中科技大学 A kind of three-dimensional super-resolution method based on the imaging of Bezier mating plate
CN109712072A (en) * 2018-12-15 2019-05-03 浙江大学 Striped illumination Fourier iteration based on total internal reflection updates super-resolution micro imaging method
CN111145089A (en) * 2019-11-18 2020-05-12 中国科学院苏州生物医学工程技术研究所 High fidelity image reconstruction method, system, computer equipment and storage medium
CN111077121A (en) * 2019-12-06 2020-04-28 中国科学院西安光学精密机械研究所 Rapid method and system for directly reconstructing structured light illumination super-resolution image in space domain
CN111127320A (en) * 2019-12-23 2020-05-08 哈尔滨工业大学（威海） Photoacoustic image super-resolution reconstruction method and device based on deep learning

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
DASHAN DONG，ET AL.: "Super-resolution fluorescence-assisted diffraction computational tomography reveals the three-dimensional landscape of the celluar organelle interactome", 《LIGHT:SCIENCE&APPLICATIONS》 *
MO YANQUAN,ET AL.: "Structured illumination microscopy artifacts caused by illumination scattering", 《RESEARCHGATE》 *
张财华等: "自适应光学在生物荧光显微成像技术中的应用", 《中国科学》 *

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115100033A (en) * 2022-05-20 2022-09-23 浙江大学 Fluorescence microscopic image super-resolution reconstruction method and device and computing equipment
CN115100033B (en) * 2022-05-20 2023-09-08 浙江大学 Fluorescent microscopic image super-resolution reconstruction method and device and computing equipment
WO2024065094A1 (en) * 2022-09-26 2024-04-04 广州超视计生物科技有限公司 Super-resolution microscopic imaging method and apparatus, and computer device and storage medium

Also Published As

Publication number Publication date
CN113112405B (en) 2022-04-12

Similar Documents

Publication Publication Date Title
Kuś et al. Holographic tomography: hardware and software solutions for 3D quantitative biomedical imaging
US8731272B2 (en) Computational adaptive optics for interferometric synthetic aperture microscopy and other interferometric imaging
US20200234080A1 (en) Reducing noise in an image
CN113112405B (en) Self-adaptive correction method of super-resolution microscope image and SIM-ODT (subscriber identity module-ODT) bimodal system
Karpinsky et al. High-resolution, real-time 3D imaging with fringe analysis
Smith et al. Structured illumination microscopy with noise-controlled image reconstructions
Ralston et al. Deconvolution methods for mitigation of transverse blurring in optical coherence tomography
JP6112872B2 (en) Imaging system, image processing method, and imaging apparatus
CN111292386B (en) CT projection metal trace complement metal artifact correction method based on U-net
Sheng et al. A constrained variable projection reconstruction method for photoacoustic computed tomography without accurate knowledge of transducer responses
Ikoma et al. A convex 3D deconvolution algorithm for low photon count fluorescence imaging
Shen et al. Complex amplitude reconstruction by iterative amplitude-phase retrieval algorithm with reference
Lee et al. High-throughput volumetric adaptive optical imaging using compressed time-reversal matrix
EP3830628B1 (en) Device and process for capturing microscopic plenoptic images with turbulence attenuation
US20200110017A1 (en) Method for determining parameters of a particle
Zuo et al. Spectral crosstalk in photoacoustic computed tomography
Kim et al. Deep learning alignment of bidirectional raster scanning in high speed photoacoustic microscopy
Ning et al. Deep self-learning enables fast, high-fidelity isotropic resolution restoration for volumetric fluorescence microscopy
Wang et al. Approximate back‐projection method for improving lateral resolution in circular‐scanning‐based photoacoustic tomography
Ruan et al. Adaptive total variation based autofocusing strategy in ptychography
Bai et al. Compressed blind deconvolution and denoising for complementary beam subtraction light-sheet fluorescence microscopy
Kang et al. Mapping nanoscale topographic features in thick tissues with speckle diffraction tomography
Kharitonov et al. Single-shot ptychography at a soft X-ray free-electron laser
KR102154649B1 (en) High speed imaging system for measuring target object within sample
Wang et al. Back-projection algorithm in generalized form for circular-scanning-based photoacoustic tomography with improved tangential resolution

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant