CN113962863A - Multi-LED multiplexing 3D-FPM reconstruction algorithm based on multilayer diffraction model - Google Patents
Multi-LED multiplexing 3D-FPM reconstruction algorithm based on multilayer diffraction model Download PDFInfo
- Publication number
- CN113962863A CN113962863A CN202111281435.4A CN202111281435A CN113962863A CN 113962863 A CN113962863 A CN 113962863A CN 202111281435 A CN202111281435 A CN 202111281435A CN 113962863 A CN113962863 A CN 113962863A
- Authority
- CN
- China
- Prior art keywords
- layer
- function
- sample
- thin sample
- complex amplitude
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000004422 calculation algorithm Methods 0.000 title claims abstract description 46
- 210000001747 pupil Anatomy 0.000 claims abstract description 52
- 238000011478 gradient descent method Methods 0.000 claims abstract description 11
- 238000005286 illumination Methods 0.000 claims description 16
- 238000004088 simulation Methods 0.000 claims description 12
- 239000013598 vector Substances 0.000 claims description 7
- 238000004613 tight binding model Methods 0.000 claims description 3
- 230000001427 coherent effect Effects 0.000 claims 1
- 230000004075 alteration Effects 0.000 abstract description 31
- 238000003384 imaging method Methods 0.000 abstract description 11
- 238000000034 method Methods 0.000 abstract description 8
- 239000000523 sample Substances 0.000 description 167
- 230000000694 effects Effects 0.000 description 9
- 230000008439 repair process Effects 0.000 description 8
- 230000005540 biological transmission Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000000386 microscopy Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 239000000126 substance Substances 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 3
- 238000000338 in vitro Methods 0.000 description 3
- 230000014509 gene expression Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000012472 biological sample Substances 0.000 description 1
- 238000004587 chromatography analysis Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000009795 derivation Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000000399 optical microscopy Methods 0.000 description 1
- 238000010186 staining Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000003325 tomography Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
- G06T3/4061—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution by injecting details from different spectral ranges
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
- G01N21/41—Refractivity; Phase-affecting properties, e.g. optical path length
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4084—Scaling of whole images or parts thereof, e.g. expanding or contracting in the transform domain, e.g. fast Fourier transform [FFT] domain scaling
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Spectroscopy & Molecular Physics (AREA)
- Chemical & Material Sciences (AREA)
- Analytical Chemistry (AREA)
- Biochemistry (AREA)
- General Health & Medical Sciences (AREA)
- Immunology (AREA)
- Pathology (AREA)
- Investigating Or Analysing Materials By Optical Means (AREA)
Abstract
A multi-LED multiplexing 3D-FPM reconstruction algorithm based on a multilayer diffraction model relates to the technical field of computational imaging, and solves the problem that the acquisition rate is too slow when a plurality of LEDs are illuminated in the prior art; updating the sample refractive index function of the thin sample layer by using a gradient descent method and updating the phase part of the latest pupil function by using a gradient descent method by using the latest loss function, the latest incident light complex amplitude of the thin sample layer and the latest sample refractive index function of the thin sample layer; and step three, judging whether the times of finishing the step two reach the preset value, if so, outputting a sample refractive index function, and otherwise, performing the step one again according to the updated result. The method greatly reduces the number of LR pictures to be acquired, effectively reduces the influence of system aberration on the reconstruction result, and improves the reconstruction precision of the algorithm.
Description
Technical Field
The invention relates to the technical field of computational imaging, in particular to a multi-LED multiplexing 3D-FPM reconstruction algorithm based on a multilayer diffraction model.
Background
Fourier stacked microscopy (FPM) is a newly developed imaging method aimed at circumventing the limitations of the Spatial Bandwidth Product (SBP) and obtaining complex images with wide field of view and high resolution. Since 2013, the Fourier laminated imaging technology is applied to the fields of optical microscopy, biomedicine, life science and the like, and a large-field-of-view and high-resolution microscopic image is obtained. Meanwhile, the 3-dimensional reconstruction of a thick sample is also advanced to a certain extent, but for the 3-dimensional reconstruction, the number of LEDs of a light source is greatly increased, which results in that the acquisition time of the system is too long, the system is not suitable for observation of living samples such as biological cells cultured in vitro, and only static staining samples or other non-biological samples can be observed at present.
Disclosure of Invention
In order to solve the problem that the acquisition rate is too low due to the fact that the number of light source LEDs is large in 3-dimensional reconstruction of Fourier laminated imaging, the invention provides a multi-LED multiplexing 3D-FPM reconstruction algorithm based on a multilayer diffraction model under a multi-LED multiplexing coding illumination strategy (namely, a plurality of LEDs are simultaneously lightened for illumination, and the number of required pictures is reduced under the condition that the quantity of acquired information is consistent).
The technical scheme adopted by the invention for solving the technical problem is as follows:
a multi-LED multiplexing 3D-FPM reconstruction algorithm based on a multilayer diffraction model comprises the following steps:
step one, constructing a loss function between a low-resolution picture obtained by a multilayer diffraction model and a low-resolution picture actually acquired by a camera; the low-resolution pictures actually acquired by the camera are a plurality of low-resolution pictures;
based on a multilayer diffraction model, the complex amplitude of emergent light of the nth layer of thin sample layer can be calculated according to the sample refractive index function of the nth layer of thin sample layer and the complex amplitude of incident light of the nth layer of thin sample layer, and a low-resolution picture received on the camera phase plane of the FPM system can be obtained through the pupil function of the FPM system and the simulation of the complex amplitude of emergent light of a thick sample; the thick sample is formed by sequentially overlapping N thin sample layers from top to bottom, N is an integer larger than 2, N is an integer and belongs to [1, N ], and a sample refractive index function of each thin sample layer can be calculated according to the multilayer diffraction model;
step two, performing the following steps on all low-resolution pictures actually acquired by the camera in the step one: updating the sample refractive index function of the thin sample layer from the Nth layer to the 1 st layer by a gradient descent method according to the latest loss function, the latest incident light complex amplitude of the thin sample layer and the latest sample refractive index function of the thin sample layer, and updating the phase part of the latest pupil function by the gradient descent method;
and step three, judging whether the number of times of finishing the step two exceeds a preset number of times or whether the loss function variable quantity obtained in the step two is smaller than a preset threshold value, if the number of times of finishing the step two exceeds the preset number of times or the loss function variable quantity is smaller than the preset threshold value, outputting an updated sample refractive index function of each thin sample layer corresponding to each low-resolution picture, and otherwise, taking the sample refractive index function of the updated thin sample layer corresponding to each low-resolution picture as the sample refractive index function of the thin sample layer in the step one and taking the updated pupil function as the pupil function in the step one to perform the step one again.
The invention has the beneficial effects that:
the invention provides a multi-LED multiplexing 3D-FPM reconstruction algorithm based on a multilayer diffraction model, which can greatly reduce the number of LR pictures required to be acquired on the basis of ensuring the reconstruction accuracy, namely, the time required by image acquisition is reduced, the acquisition speed is improved, and meanwhile, the influence of system aberration on the reconstruction result can be effectively reduced in the pupil function repair process embedded in the algorithm, so that the reconstruction accuracy of the algorithm is further improved. By adopting the multi-LED multiplexing 3D-FPM reconstruction algorithm based on the multilayer diffraction model, the FPM system is not only suitable for observing living samples such as biological cells cultured in vitro, but also has better observation effect.
Drawings
FIG. 1 is an X-Y, X-Z, Y-Z cross-sectional view and a single LED reconstruction result comparison graph of a real 3D sample simulated by a multi-LED multiplexing 3D-FPM reconstruction algorithm based on a multilayer diffraction model.
Fig. 2 is a graph comparing the reconstruction results with and without the influence of aberrations, single LED, multiple LED, and without pupil repair.
Fig. 3 is a perspective contrast diagram of the reconstruction results with or without the influence of aberrations, single LED, multiple LEDs, and without pupil repair.
FIG. 4 is a comparison graph of the system aberration reconstructed by the present invention and the system aberration preset by simulation.
Figure 5 is a graph comparing the results of a multi-LED reconstruction without aberration effects, with aberration effects, and with aberration effects and the addition of a pupil function.
Detailed Description
In order that the above objects, features and advantages of the present invention can be more clearly understood, a more particular description of the invention will be rendered by reference to the appended drawings.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, however, the present invention may be practiced in other ways than those specifically described herein, and therefore the scope of the present invention is not limited by the specific embodiments disclosed below.
Multi-LED multiplexing 3D-FPM reconstruction algorithm based on multilayer diffraction model
The invention provides a 3D reconstruction algorithm combining multi-LED coding illumination, and meanwhile, a repair process for system aberration is embedded in the algorithm, so that the influence of the system aberration on a reconstruction result can be eliminated, and meanwhile, the number of required low-resolution pictures is reduced on the premise of ensuring reconstruction accuracy, and the specific flow of the algorithm is as follows:
step one, constructing a loss function between a low-resolution picture obtained by a multilayer diffraction model and a low-resolution picture actually acquired by a camera, and performing step two; the low-resolution pictures actually acquired by the camera are a plurality of low-resolution pictures, and the low-resolution pictures obtained by the multilayer diffraction model correspond to the low-resolution pictures actually acquired by the camera one by one;
based on a multilayer diffraction model, the complex amplitude of emergent light of the nth layer of thin sample layer can be calculated according to the sample refractive index function of the nth layer of thin sample layer and the complex amplitude of incident light of the nth layer of thin sample layer, and a low-resolution picture received on the camera phase plane of the FPM system can be obtained through the pupil function of the FPM system and the simulation of the complex amplitude of emergent light of a thick sample; the thick sample is formed by sequentially overlapping N thin sample layers from top to bottom, N is an integer larger than 2, N is an integer and belongs to [1, N ], and a sample refractive index function of each thin sample layer can be calculated according to the multilayer diffraction model. The complex amplitude of emergent light of the a-th layer of thin sample layer is equal to the complex amplitude of incident light of the a + 1-th layer of thin sample layer, a is an integer and belongs to [1, N-1 ].
Step two, performing the following steps on all low-resolution pictures actually acquired by the camera in the step one, and then performing the step three: and updating the sample refractive index function of the thin sample layer corresponding to the low resolution picture from the thin sample layer from the Nth layer to the 1 st layer by using the latest loss function, the latest thin sample layer incident light complex amplitude corresponding to the low resolution picture and the latest thin sample layer sample refractive index function corresponding to the low resolution picture by using a gradient descent method to obtain the updated sample refractive index function of the thin sample layer, and updating the phase part of the latest pupil function corresponding to the low resolution picture by using a gradient descent method to obtain the updated pupil function corresponding to the low resolution picture.
Step three, judging whether the number of times of finishing the step two exceeds a preset number of times or whether the loss function variable quantity obtained in the step two is smaller than a preset threshold, if the number of times of finishing the step two exceeds the preset number of times or the loss function variable quantity is smaller than the preset threshold, outputting an updated sample refractive index function of each thin sample layer corresponding to each low-resolution picture, and finishing the operation; otherwise, updating the multilayer diffraction model in the first step by the sample refractive index of the thin sample layer updated corresponding to each low-resolution picture and the pupil function updated corresponding to each low-resolution picture, and executing the first step again.
The multi-LED multiplexing 3D-FPM reconstruction algorithm based on the multilayer diffraction model is detailed below.
And deducing a multilayer diffraction model of the thick sample according to Helmholtz equation, wherein the multilayer diffraction model comprises calculation of complex amplitude of emergent light of each thin sample layer, calculation of a sample refractive index function of each thin sample layer and simulation of low-resolution complex amplitude received on a camera phase plane of the FPM system, and the multilayer diffraction model simulates the complex amplitude of emergent light of the thick sample (namely the complex amplitude of emergent light of the last thin sample layer) through a pupil function of the Fourier laminated micro-imaging system to obtain the low-resolution complex amplitude (namely a low-resolution picture) received on the camera phase plane. The method specifically comprises the following steps: firstly, an equation of scattered light complex amplitude of a thick sample relative to a sample refractive index function is deduced based on a Helmholtz equation, an equation of scattered light complex amplitude of a thin sample layer is deduced according to the equation of scattered light of the thick sample relative to the sample refractive index function, an equation of emergent light complex amplitude of the thin sample layer is deduced according to a Born approximate condition and the equation of scattered light complex amplitude of the thin sample layer, and a low-resolution picture received on a phase plane of an FPM system camera is deduced according to the equation of emergent light complex amplitude of the thin sample layer and a pupil function.
The thick sample is formed by sequentially stacking a plurality of thin sample layers from top to bottom, wherein the traversal of the thin sample layers from top to bottom is referred to as forward direction, and the traversal from bottom to top is referred to as reverse direction. Samples broadly refer to thick samples and thin sample layers.
Firstly, a multilayer diffraction chromatography model (namely a multilayer diffraction model) for a thick sample is constructed as a basic model in the algorithm, the model considers the superposition of a thick sample (the thickness of the thin sample layer is less than 1um, preferably the thickness of the middle thin sample layer is 400-500 nm, and the thickness of the thin sample layer in the embodiment is about 450 nm) of a plurality of thin sample layers (the thickness of the thin sample layer is less than 1um, and the thickness of the middle thin sample layer is about 450 nm), the thin sample layers are equal in thickness, the thickness of each thin sample layer is set to be delta z, N layers are totally included, N is an integer greater than 2, and emergent light of the previous thin sample layer is incident light of the next thin sample layer. For theoretical analysis using diffraction tomography for each thin sample layer, equation (1) for the sample scattered light as a function of the sample refractive index is first derived based on the helmholtz equation, and the present invention primarily considers the Born approximation (Born approximation), for which the Rytov approximation (ritov approximation) and the Born approximation equation are substantially the same. The scattering light equation for thick samples was derived based on the helmholtz equation as follows:
wherein U iss(r, Z) represents the complex amplitude of the scattered light, r represents XY plane coordinates at the scattered light field, Z represents Z-axis space coordinates at the scattered light field, r 'represents XY plane coordinates on the integral path of the incident light to the scattered light, Z' represents Z-axis space coordinates on the integral path of the incident light to the scattered light, and further r-r 'and Z-Z' represent the distances between the scattered light coordinates and the integral coordinates on the XY plane and the Z axis, respectively. G (r-r ', z-z ') represents the Green's function in the integral, V (r ', z ') represents the refractive index function of the sample at (r ', z '), Uin(r ', z') represents the complex amplitude of the incident light at (r ', z'), and concrete expressions of G (r-r ', z-z') and V (r ', z') are as follows:
wherein the content of the first and second substances,
representing wave vectors in the sample, n0Is the refractive index of the medium surrounding the sample, λ represents the LED illumination wavelength, n (·,) represents the complex refractive index function of the sample, i.e., n (r ', z') represents the complex refractive index function of the sample at (r ', z'), kxyRepresents the wave vector component of the scattered light in the XY plane direction,
and (2) representing wave vector components of scattered light in the Z-axis direction, and further simplifying the formula (1) to obtain an equation of complex amplitude of the scattered light of each thin sample layer with respect to a refractive index function V (r, Z) of the thin sample layer with the thickness deltaz, wherein the complex amplitude of the scattered light of the thin sample layer is represented as follows:
wherein the content of the first and second substances,
denotes the complex amplitude of scattered light of the n-th thin sample layer, k for conveniencexyDenoted by u, F denotes a two-dimensional fourier transform;
representing the two-dimensional Fourier transform of the Green function G (r-r ', z-z') in equation (2),
denotes the complex amplitude of incident light for the n-th thin sample layer, and V (r, n · Δ z) denotes the sample refractive index function for the n-th thin sample layer.
Finally, considering the Bom approximation condition, the complex amplitude of the emergent light of the n-th thin sample layer can be expressed as the complex amplitude of the incident light of the n-th thin sample layer after transmitting the distance Δ z in the thick sample and the scattered lightComplex amplitude of vibration
And the emergent light of the nth layer is equivalent to the incident light of the (n + 1) th layer, and the complex amplitude of the emergent light of the nth layer thin sample layer is as follows:
since the plane conjugated with the phase plane in the fourier stacked microscopy imaging system is not the last layer of the sample but a layer in the middle of the thick sample (a thin sample layer corresponding to the middle thickness of the thick sample), the intermediate layer to which the emergent light of the thin sample layer in the last layer is transmitted in the reverse direction is needed, that is, a distance is
The inverse transmission factor of (b) is combined with a pupil function of a fourier stacked microscopy imaging system to obtain a low-resolution complex amplitude u (r) received on a camera phase plane, where u (r) is:
that is, the image taken by the camera can be simulated by the formula (3), and the picture actually acquired by the camera is called a low-resolution picture. Wherein P (k)xy) As a pupil function of a fourier stack microscopy imaging system,
and the complex amplitude of emergent light of the last thin sample layer is shown, and the last thin sample layer is the thin sample layer closest to the camera in the thick sample.
And (5) constructing a loss function according to the multilayer diffraction model, wherein the formula (4) and the formula (5) are the obtained multilayer diffraction model. And updating the sample refractive index function of the thin sample layer from the Nth layer to the 1 st layer by using the loss function, the incident light complex amplitude of each thin sample layer and the sample refractive index function of each thin sample layer and adopting a gradient descent method.
That is, the multilayer diffraction model and the picture acquired by the actual FPM system for the thick sample need to be combined to perform inverse update reconstruction on the 3D sample refractive index function V (r, n Δ z) (the sample refractive index function of the nth thin sample layer), so that a loss function used by the reconstruction algorithm needs to be constructed first, and for the case of multi-LED illumination, a loss function between the low-resolution picture acquired by constructing the multilayer diffraction model and the actually acquired low-resolution picture is constructed, that is, the loss function H adopted by the reconstruction algorithm of the present invention is as follows:
wherein, T represents that T LR pictures (LR represents low resolution) are shot by a camera in total under the multi-LED multiplexing illumination strategy, and T is the serial number of the T LR pictures, i.e. It(r) represents the light intensity of the T-th LR picture actually acquired by the camera, T, T and m are positive integers, T is more than 2, and LtShowing the LED sequence required for taking the t-th LR picture, m showing the m-th LED in the LED sequence, Um(r) represents the low resolution complex amplitude under the illumination of the mth LED as simulated by the above multilayer diffraction model equation (5),
and (3) adopting a gradient descending updating algorithm to obtain the derivative of the loss function H with respect to the sample refractive index function of the thin sample layer from the Nth layer to the 1 st layer. And (4) deriving the loss function H with respect to the sample refractive index function of the n-th thin sample layer to obtain gradient information about the n-1 th layer, and obtaining the updated sample refractive index function of the n-1 th thin sample layer according to the gradient information of the n-1 th layer.
First consider the update process for the last (nth) thin sample layer, whose gradient from the loss function is:
for convenience of expression, the formula is exemplified by the t-th LR diagram, where VNThe refractive index function of the sample of the thin sample layer of the Nth layer is expressed by the formula (5)
The expansion is as follows:
wherein, the upper right corner of a certain symbol represents a conjugate, P*(u) is the conjugate of P (u).
Equation (8) is obtained in conjunction with equation (4):
wherein the content of the first and second substances,
to represent
The conjugate of (a) to (b),
to represent
The conjugate of (a) to (b),
represents the complex amplitude of incident light on the Nth layer, i.e., the complex amplitude of outgoing light on the N-1 th layer obtained by equation (4)
That is, the algorithm needs to store the complex amplitude of the incident light of each layer in the forward simulation process in each iteration (after one step two is completed, one iteration is defined to be completed), so that the update gradient of the last layer of sample under the multiplexing of multiple LEDs can be obtained by combining the formula (7), the formula (8) and the formula (9), and the update gradient is obtained according to the formula (7), the formula (8) and the formula (9)
The average result can further obtain the updated refractive index function of the Nth layer sample
Considering next the updating gradient of the thin sample layer of the N-1 th layer, because the function of the thin sample layer of the N-1 th layer is between the emergent light of the thick sample and the thin sample layer of the N-1 th layer, namely the updating result of the thin sample layer of the N-1 th layer should be in the updating of the sample refractive index function of the thin sample layer of the N-1 th layer
I.e., the updated gradient of the sample refractive index function for the N-1 th thin sample layer is expressed as follows:
wherein, VN-1Represents the sample refractive index function of the (N-1) th thin sample layer,
to represent
The conjugate of (a) to (b),
represents the complex amplitude of incident light of the (N-1) th layer, i.e., the complex amplitude of emergent light of the (N-2) th layer obtained by the formula (4)
Wherein
Expressed as:
wherein the content of the first and second substances,
is that
Wherein the combination of formula (7) and formula (8) can be obtained:
that is, gradient information about the N-1 th layer is obtained, and then the updated sample refractive index function of the thin sample layer of the N-1 th layer can be obtained
And meanwhile, sequentially analogizing to the first thin sample layer and updating to the first layer all the time, wherein the updated sample refractive index function of the last thin sample layer and the incident light complex amplitude of the layer obtained by forward simulation are required to be used for updating each layer.
The phase part of the pupil function of the FPM system is then updated with gradient descent, i.e. only the influence of the aberrations of the system on the reconstruction is taken into account.
Meanwhile, the influence of the aberration of the Fourier laminated microscopic imaging system on the reconstruction result is considered, and the aberration corresponds to the phase part in the pupil function, so that the repair process of the pupil function is inserted into the algorithm, and the specific formula is as follows:
wherein, Pq(kxy) Representing the complex amplitude, P, of the original pupil functionq+1(kxy) Represents according to Pq(kxy) The complex amplitude of the pupil function obtained by the update, α represents the update step for the pupil function, and δ represents the regularization small quantity that prevents the denominator from appearing zero in equation (13), where W isn，m(kxy) Refer to in formula (5)
This section.
The updating process of the pupil function and the updating process of the sample refractive index function of the thin sample layer are alternately carried out, namely, the pupil function is updated once after all the thin sample layers corresponding to each low-resolution picture are updated. The updating of the pupil function adds a complex amplitude constraint to the pupil function to speed its convergence.
According to the method, T low-resolution pictures need to be acquired, the sample refractive index functions and the pupil functions of all the thin sample layers are updated in two steps for each acquired LR picture, the LR pictures are used for updating and then are regarded as one iteration, when the iteration times exceed the preset value or the loss function variation is small, the algorithm is stopped, the reconstructed sample 3D refractive index function is output, and finally the updated sample refractive index function of each thin sample layer corresponding to each low-resolution picture is output. Based on the prior art, the high-resolution picture corresponding to each low-resolution picture can be obtained according to the updated sample refractive index function of each thin sample layer corresponding to each low-resolution picture obtained by the method.
Specific application examples of the present invention are described in detail below.
Firstly, according to a preset multi-LED multiplexing strategy, LEDs in an LED array light source are sequentially lightened, and low-resolution images corresponding to different multiplexing LEDs under illumination are collected on a camera through an FPM system.
Secondly, according to the current parameters of the FPM system, setting a wave vector and a pupil function P (u) corresponding to each LED in a program, wherein an initial pupil function is assumed to be a Coherence Transfer Function (CTF) of the FPM system, and the initial pupil function is shown as the following formula:
wherein NA is the aperture of the FPM system objective lens in the vertical direction, and λ is the used illumination wavelength.
And simultaneously setting the three-dimensional size of the overall 3D refractive index function of the reconstructed sample, including the total number of thin sample layers and the two-dimensional size of each thin sample layer, and setting the value of the three-dimensional size to 0 when the multi-LED multiplexing 3D-FPM reconstruction algorithm starts.
The 3D refractive index function of the initially assumed sample is subjected to forward transmission through a Born approximate multilayer diffraction model, the complex amplitude of emergent light of each thin sample layer under the transmission is obtained by using a formula (4), and the complex amplitude is recorded for later reverse gradient derivation.
When the transmission process goes to the last layer, the simulated low-resolution images corresponding to different LED illuminations under the multilayer diffraction model are obtained by using the formula (5). Meanwhile, combining a multiplexing strategy under multi-LED illumination to obtain a simulated low-resolution image obtained under corresponding single multiplexing illumination:
wherein L istIs a list of LED numbers lit under a single multiplexed illumination.
Constructing a loss function between a low-resolution picture obtained through multilayer diffraction model simulation and an actually acquired low-resolution picture
And updating the refractive index function of the sample from the Nth layer to the 1 st layer of the thin sample layers by adopting a gradient descent method according to the loss function, the incident light complex amplitude of each thin sample layer and the sample refractive index function of each thin sample layer.
The updating of the pupil function and the updating of the sample are alternated, while amplitude constraints on the pupil function are added to speed up its convergence. The refractive index function and pupil function of all thin sample layers of the thick sample are updated for each LR picture with two steps and considered as one iteration after all LR pictures are used for updating. The sample 3D refractive index function (i.e., the refractive index function for each thin sample layer) and the pupil function after this iteration are saved and returned to equation (4), the new sample 3D refractive index function is taken as the initial V (r, n · Δ z), equation (4) is recalculated, equation (5) is recalculated using the new pupil function, and then the loss function construction is performed. And finally, when the iteration times exceed the preset times or the loss function is not reduced or the change tends to be stable (the change amount is smaller than a preset threshold), outputting the sample 3D refractive index function after current iterative reconstruction and the updated pupil function.
The fourier stacked imaging technique used in the present invention is a classical FPM system architecture, and the system parameters are set to include a 40X objective lens with NA 0.6, a CCD sensor (i.e. the camera described above) with a pixel size of 6.5 μm and a programmable LED matrix,
FIG. 1 is a cross-sectional diagram of X-Y, X-Z, Y-Z cross-section of a real 3D sample refractive index function simulated by the algorithm of the present invention and a comparison of results after reconstruction by a single LED (using 187 low resolution pictures), wherein the left half is a simulation true value which is a processing result of a multilayer diffraction model, and the right half is a reconstruction result obtained after calculation by the reconstruction algorithm.
Fig. 2 is a comparison of a single LED reconstruction result without aberration, a reconstruction result without aberration using multi-LED multiplexing, a multi-LED reconstruction result with aberration, and a multi-LED multiplexed reconstruction result with aberration and when a pupil repair algorithm as mentioned in the present invention is added, wherein the upper left, upper right, lower left, and lower right of each of the four pictures respectively represent an X-Y cross-sectional view, an X-Z cross-sectional view, a Y-Z cross-sectional view, and a first layer refractive index function of the reconstruction result refractive index function.
Figure 3 is a 3D result perspective of a single LED reconstruction result without the effect of aberrations, a reconstruction result with multiple LED multiplexing without the effect of aberrations, a multiple LED reconstruction result with the effect of aberrations, and a multiple LED multiplexed reconstruction result with the effect of aberrations and incorporating the pupil repair algorithm referred to in the present invention.
FIG. 4 is a comparison between the system aberration reconstructed by the present invention and the system aberration preset in the simulation, in which the abscissa and ordinate in the figure are the size of the reconstructed aberration, and the color change bar is the aberration PV value of the aberration diagram, i.e. the unit is 2 π.
Fig. 5 is a graph comparison of a reconstruction result under the condition of multi-LED multiplexing mentioned in the present invention without the influence of aberration, a multi-LED reconstruction result under the influence of aberration, and a multi-LED multiplexing reconstruction result under the influence of aberration when a pupil restoration algorithm is added, wherein the abscissa is the iteration number of the algorithm, and the ordinate is the root mean square error value between a low resolution picture obtained by each iteration simulation of the algorithm and an actually measured low resolution picture.
As can be seen from fig. 1 to 5, the multi-LED multiplexing 3D-FPM reconstruction algorithm based on the multilayer diffraction model provided in the present invention can greatly reduce the number of LR pictures to be acquired, i.e. reduce the time required for image acquisition, and improve the acquisition speed, on the basis of ensuring the reconstruction accuracy, and meanwhile, the pupil function repair process embedded in the algorithm can effectively reduce the influence of system aberration on the reconstruction result, and further improve the reconstruction accuracy of the algorithm. By adopting the multi-LED multiplexing 3D-FPM reconstruction algorithm based on the multilayer diffraction model, the FPM system is not only suitable for observing living samples such as biological cells cultured in vitro, but also has better observation effect.
Claims (10)
1. The multi-LED multiplexing 3D-FPM reconstruction algorithm based on the multilayer diffraction model is characterized by comprising the following steps:
step one, constructing a loss function between a low-resolution picture obtained by a multilayer diffraction model and a low-resolution picture actually acquired by a camera; the low-resolution pictures actually acquired by the camera are a plurality of low-resolution pictures;
based on a multilayer diffraction model, the complex amplitude of emergent light of the nth layer of thin sample layer can be calculated according to the sample refractive index function of the nth layer of thin sample layer and the complex amplitude of incident light of the nth layer of thin sample layer, and a low-resolution picture received on the camera phase plane of the FPM system can be obtained through the pupil function of the FPM system and the simulation of the complex amplitude of emergent light of a thick sample; the thick sample is formed by sequentially overlapping N thin sample layers from top to bottom, N is an integer larger than 2, N is an integer and belongs to [1, N ], and a sample refractive index function of each thin sample layer can be calculated according to the multilayer diffraction model;
step two, performing the following steps on all low-resolution pictures actually acquired by the camera in the step one: updating the sample refractive index function of the thin sample layer from the Nth layer to the 1 st layer by a gradient descent method according to the latest loss function, the latest incident light complex amplitude of the thin sample layer and the latest sample refractive index function of the thin sample layer, and updating the phase part of the latest pupil function by the gradient descent method;
and step three, judging whether the number of times of finishing the step two exceeds a preset number of times or whether the loss function variable quantity obtained in the step two is smaller than a preset threshold value, if the number of times of finishing the step two exceeds the preset number of times or the loss function variable quantity is smaller than the preset threshold value, outputting an updated sample refractive index function of each thin sample layer corresponding to each low-resolution picture, and otherwise, taking the sample refractive index function of the updated thin sample layer corresponding to each low-resolution picture as the sample refractive index function of the thin sample layer in the step one and taking the updated pupil function as the pupil function in the step one to perform the step one again.
2. The multi-LED multiplexed 3D-FPM reconstruction algorithm based on the multilayer diffraction model as claimed in claim 1, wherein the complex amplitude of the emergent light of the n-th thin sample layer is equal to the sum of the complex amplitude of the incident light of the n-th thin sample layer after being transmitted by the thickness of the n-th thin sample layer and the complex amplitude of the scattered light of the n-th thin sample layer.
3. The multi-LED multiplexed 3D-FPM reconstruction algorithm based on multi-layered diffraction model of claim 1, wherein the multi-layered diffraction model is constructed by:
step 0.1, deducing a function of the complex amplitude of the scattered light of the thick sample on a Green function and a sample refractive index function according to a Helmholtz equation;
step 0.2, deducing the complex amplitude of the scattered light of the thin sample layer according to the complex amplitude of the scattered light of the thick sample;
step 0.3, deriving the complex amplitude of emergent light of the nth layer of thin sample layer according to the Bom approximate condition and the complex amplitude of scattered light of the nth layer of thin sample layer;
and 0.4, deducing a low-resolution picture received on a camera phase plane according to the complex amplitude of emergent light of the Nth layer of thin sample layer and a pupil function of the FPM system.
4. The multi-LED multiplexed 3D-FPM reconstruction algorithm based on the multilayer diffraction model as claimed in claim 1, wherein the complex amplitude of the emergent light of the nth thin sample layer is:
the low resolution complex amplitude received on the camera phase plane is:
where r denotes the XY plane coordinate at the scattered light field, Δ z denotes the thickness of the thin sample layer, F denotes the two-dimensional fourier transform, and u ═ kxyDenotes the wave vector component, k, of the scattered light in the XY plane directionzRepresenting the wave-vector component of the scattered light in the Z-axis direction,
denotes the complex amplitude of incident light of the n-th thin sample layer, V (r, n.DELTA.z) denotes the sample refractive index function of the n-th thin sample layer, P (k)xy) As a function of the pupil of the FPM system,
representing the complex amplitude of the emerging light from the last thin sample layer.
5. The multi-LED multiplexed 3D-FPM reconstruction algorithm based on multilayer diffraction model of claim 4, wherein the scattered light of the thick sample is:
the complex amplitude of scattered light of the nth thin sample layer is:
wherein Z represents a Z-axis spatial coordinate at the scattered light field, r 'represents an XY-plane coordinate on an integral path of the incident light to the scattered light, Z' represents a Z-axis spatial coordinate on an integral path of the incident light to the scattered light, G (r-r ', Z-Z') represents a Green function in the integral, V (r ', Z') represents a sample refractive index function at (r ', Z'), Um(r ', z') represents the complex amplitude of the incident light at (r ', z').
6. The multi-LED multiplexed 3D-FPM reconstruction algorithm based on multilayer diffraction model of claim 1, wherein the step of updating the sample refractive index function of the thin sample layer from the N-th layer to the 1 st layer using the gradient descent method comprises: and (4) deriving the loss function with respect to the sample refractive index function of the n-th thin sample layer to obtain gradient information about the n-1 th layer, and obtaining the updated sample refractive index function of the n-1 th thin sample layer according to the gradient information of the n-1 th layer.
7. The multi-LED multiplexed 3D-FPM reconstruction algorithm based on multilayer diffraction model of claim 1, wherein the loss function H is:
wherein T represents that a camera is required to actually acquire T low-resolution pictures in total under the multi-LED multiplexing illumination strategy, T is the serial number of the T low-resolution pictures, and It(r) represents the light intensity of the T-th low-resolution picture actually acquired by the camera, T, T and m are positive integers, T is more than 2, and LtThe LED sequence required by the t-th low-resolution picture actually acquired by the camera is represented, m represents the m-th LED, U in the LED sequencem(r) represents the low resolution complex amplitude under illumination of the mth LED obtained by multilayer diffraction model simulation,
8. the multi-LED multiplexed 3D-FPM reconstruction algorithm based on multilayer diffraction model of claim 7, wherein the pupil function is updated by the formula:
wherein, Pq(kxy) Representing complex amplitude, P, of the original pupil functionq+1(kxy) Represents according to Pq(kxy) The complex amplitude of the pupil function obtained by updating, q represents the number of times of completing the second step, alpha represents the updating step length of the pupil function, delta represents the regularization small quantity for preventing the denominator from generating zero value, and Wn，m(kxy) Finger-shaped
9. The multi-LED multiplexed 3D-FPM reconstruction algorithm based on multilayer diffraction model of claim 1, wherein the first performing step one, the pupil function is:
wherein k isxyAnd the wave vector components of the scattered light in the XY plane direction are shown, CTF is a coherent transfer function of the FPM system, NA is the aperture of an objective lens of the FPM system in the vertical direction, and lambda is the illumination wavelength of the LED.
10. The multi-LED multiplexed 3D-FPM reconstruction algorithm based on multilayer diffraction model of claim 1, wherein the thin sample layers are the same thickness.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111281435.4A CN113962863B (en) | 2021-11-01 | 2021-11-01 | Multi-LED multiplexing 3D-FPM reconstruction algorithm based on multilayer diffraction model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111281435.4A CN113962863B (en) | 2021-11-01 | 2021-11-01 | Multi-LED multiplexing 3D-FPM reconstruction algorithm based on multilayer diffraction model |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113962863A true CN113962863A (en) | 2022-01-21 |
| CN113962863B CN113962863B (en) | 2024-04-09 |
Family
ID=79468743
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111281435.4A Active CN113962863B (en) | 2021-11-01 | 2021-11-01 | Multi-LED multiplexing 3D-FPM reconstruction algorithm based on multilayer diffraction model |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113962863B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150317508A1 (en) * | 2014-04-30 | 2015-11-05 | University Of Connecticut | Methods and Systems for Fourier Ptychographic Imaging |
| US20150331228A1 (en) * | 2014-05-13 | 2015-11-19 | California Institute Of Technology | Ptychography imaging systems and methods with convex relaxation |
| CN107272177A (en) * | 2017-07-04 | 2017-10-20 | 清华大学 | Micro imaging system |
| US20180088309A1 (en) * | 2012-10-30 | 2018-03-29 | California Institute Of Technology | Embedded pupil function recovery for fourier ptychographic imaging devices |
| CN108550108A (en) * | 2017-09-28 | 2018-09-18 | 武汉大学 | A kind of Fourier's lamination image method for reconstructing minimized based on phase iteration |
| CN110378981A (en) * | 2019-07-19 | 2019-10-25 | 中国科学院长春光学精密机械与物理研究所 | Fourier's lamination microscope pupil restoration methods neural network based |
-
2021
- 2021-11-01 CN CN202111281435.4A patent/CN113962863B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180088309A1 (en) * | 2012-10-30 | 2018-03-29 | California Institute Of Technology | Embedded pupil function recovery for fourier ptychographic imaging devices |
| US20150317508A1 (en) * | 2014-04-30 | 2015-11-05 | University Of Connecticut | Methods and Systems for Fourier Ptychographic Imaging |
| US20150331228A1 (en) * | 2014-05-13 | 2015-11-19 | California Institute Of Technology | Ptychography imaging systems and methods with convex relaxation |
| CN107272177A (en) * | 2017-07-04 | 2017-10-20 | 清华大学 | Micro imaging system |
| CN108550108A (en) * | 2017-09-28 | 2018-09-18 | 武汉大学 | A kind of Fourier's lamination image method for reconstructing minimized based on phase iteration |
| CN110378981A (en) * | 2019-07-19 | 2019-10-25 | 中国科学院长春光学精密机械与物理研究所 | Fourier's lamination microscope pupil restoration methods neural network based |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113962863B (en) | 2024-04-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Zuo et al. | Wide-field high-resolution 3D microscopy with Fourier ptychographic diffraction tomography | |
| Pan et al. | High-resolution and large field-of-view Fourier ptychographic microscopy and its applications in biomedicine | |
| US10386302B2 (en) | Multiview light-sheet microscopy | |
| CN111429562B (en) | Wide-field color light slice microscopic imaging method based on deep learning | |
| CN110378473B (en) | Phase chromatography method and device based on deep learning and random pattern | |
| CN113711133A (en) | System and method for color holographic microscope based on deep learning | |
| Yang et al. | 3-D reconstruction of microtubules from multi-angle total internal reflection fluorescence microscopy using Bayesian framework | |
| CN114331840A (en) | Method and device for reconstructing high-fidelity super-resolution microscopic image | |
| Wang et al. | Three-dimensional refractive index reconstruction of red blood cells with one-dimensional moving based on local plane wave approximation | |
| CN114972284A (en) | Lens-free microscopic imaging system and method based on self-supervision deep learning | |
| CN114998507A (en) | Luminosity three-dimensional reconstruction method based on self-supervision learning | |
| CN113962863B (en) | Multi-LED multiplexing 3D-FPM reconstruction algorithm based on multilayer diffraction model | |
| Kikuchi et al. | Three-dimensional computed tomography for optical microscopes | |
| CN111524078B (en) | Dense network-based microscopic image deblurring method | |
| CN114881907B (en) | Optical microscopic image multi-depth-of-field focus synthesis method and system and image processing method | |
| US20220026699A1 (en) | Method and System for Acquisition of Fluorescence Images of Live-cell Biological Samples | |
| WO2022173848A1 (en) | Methods of holographic image reconstruction with phase recovery and autofocusing using recurrent neural networks | |
| CN104268929B (en) | Fluorescence microscope images 3D method for reconstructing and device based on image intensity all-order derivative | |
| Wang et al. | Improved SNR and super-resolution reconstruction of multi-scale digital holography based on deep learning | |
| Boland et al. | Improving axial resolution in SIM using deep learning | |
| CN110221421A (en) | Structured Illumination super-resolution micro imaging system and method based on machine learning | |
| WO2018230615A1 (en) | Image processing device, computer program, and image adjusting method | |
| Tong et al. | Three-dimensional refractive index microscopy based on the multi-layer propagation model with obliquity factor correction | |
| Luo | Advanced lens-free imaging techniques for complex cell structures and organoids | |
| CN115602318B (en) | Microscopic imaging method based on chromatic dispersion and application thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |