CN114529476A - Lensless holographic microscopic imaging phase recovery method based on decoupling-fusion network - Google Patents

Abstract

The application relates to the field of computer vision and microscopic imaging, and particularly provides a lens-free holographic microscopic imaging phase recovery method based on a decoupling-fusion network. The method comprises the following steps: s1, measuring parameters of the lens-free holographic microscopic imaging system to be recovered; s2, obtaining a sample and constructing a training sample set and a testing sample set; s3, constructing a phase recovery network; s4, training the constructed phase recovery network; and S5, performing phase recovery to solve complex amplitude. The method uses a decoupling network to decouple double-channel complex matrix information from a single-channel holographic brightness image, uses a fusion network to fuse multi-frame collected information, and combines the information with a Fresnel diffraction physical model, so that the learning target is clear and the interpretability is strong; the reconstruction accuracy is high, and the visual effect of the reconstructed phase difference and amplitude image is good; the phase recovery network used by the invention needs fewer gradient descending rounds and has higher phase recovery speed.

Description

Phase recovery method for lensless holographic microscopy based on decoupling-fusion network

technical field

The present application relates to the fields of computer vision and microscopic imaging, and in particular, to a lensless holographic microscopic imaging phase recovery method based on a decoupling-fusion network, which can be used for the amplitude and phase difference of a coaxial holographic lensless microscopic imaging system. Microscopic imaging.

Background technique

In medical and biological research, microscope is a commonly used research tool, which plays an extremely important role in observing pathological sections and microbial structures. However, due to its structural characteristics and imaging principles, traditional microscopes will bring problems such as incompatibility between the field of view and magnification, lens aberration and chromatic distortion, and cannot record the depth information of the sample, which is not good for translucent phase objects. . Lensless holographic microscopy imaging technology is based on the principle of optical holography. It uses a photoelectric sensor such as CMOS or CCD to be close to the sample plane, collects the brightness image of the sample in a coherent light field or a partially coherent light field, and uses an algorithm to recover the complete wavefront information of the sample plane. Technology.

In a lensless holographic microscope imaging system, the sample plane is placed close to the CMOS or CCD photoelectric sensor plane, and the axial distance between the two planes is called the defocus distance, which is usually between a few hundred microns and a few millimeters. The light source of light or partially coherent light illuminates the sample plane so that it is projected and imaged on the CMOS or CCD photoelectric sensor plane. The imaging process approximates the Fresnel diffraction model in the near-field region, and the brightness image recorded by the CMOS or CCD photoelectric sensor is in the light field. The interference hologram of the sample, from the collected single frame or multi-frame hologram, can solve the wavefront complex amplitude information including the amplitude and phase in the sample plane through the phase recovery algorithm, so as to realize the amplitude or phase contrast imaging of the sample plane. Compared with the traditional optical microscope, the lensless microscopic imaging system has the characteristics of large field of view, simple system structure, no lens distortion, aberration, etc., and the depth information can be calculated from the recovered phase information to realize three-dimensional imaging.

In the lensless holographic microscope imaging system, the common methods of using the acquired hologram brightness image to solve the phase can be divided into two types: the traditional iterative-based phase recovery method and the neural network-based phase recovery method. Recovery methods, such as the G-S phase iterative algorithm proposed by Gerchberg and Saxton in their 1972 paper titled "A practical algorithm for the determination of the phase from image and diffraction plane pictures", which randomly generates an initial phase, which is acquired with the sample plane The complex amplitude is synthesized from the luminance image of , and the complex amplitude is iteratively projected between the sample plane and the imaging plane, and the known luminance image on the two planes is used to replace the amplitude, and the phase information is corrected in the iterative process until the termination condition is satisfied. Using this method, phase information that meets certain error requirements can be obtained. The phase recovery algorithm based on iterative method proposed by some later studies is mostly improved on the G-S iterative phase recovery algorithm. This kind of iterative algorithm will fall into a local optimal solution after several rounds of iterations, resulting in a slow decrease in error, and inevitably requires a large amount of calculation, with high time cost.

With the outstanding progress of deep learning in the field of image restoration, image reconstruction and other image problems, many scholars have begun to discuss the application of neural networks to the phase restoration problem to solve many problems faced by traditional methods. At present, typical deep learning-based There are several methods of phase recovery: Yichen Wu et al. published an article entitled "Extended depth-of-field in holographic imaging using deep-learning-based autofocusing and phase recovery" in the journal "Optica" in 2018. In this paper, a deep learning-based phase recovery method is disclosed, which utilizes a U-shaped network structure to achieve end-to-end phase recovery. The method is trained on a large number of datasets with real labels, and learns the mapping relationship from the luminance image of a single frame of hologram collected at a fixed defocus distance to the complex amplitude of the sample plane. The advantages of this method are: in application, for The accuracy of accurately obtaining the defocus distance from the sample plane to the imaging plane is not high, because the method is robust within a certain defocus distance range. However, this method requires a large amount of data with real labels for training, so the workload of obtaining the dataset is huge, and the interpretability of the network is poor. In addition, the reconstruction performance on experimental data that is out of the dataset style is not good. good, with limitations in generalization performance.

Fei Wang et al. published an article titled "Phase imaging with an untrained neural network" in the journal "Light: Science & Applications" in 2020, disclosing a phase recovery method that combines a physical model to a deep neural network , this method uses an encoding and decoding network structure, the encoding end uses a U-shaped network to generate the prediction of the complex amplitude, and the decoding end uses the known Fresnel diffraction physical model to forward the complex amplitude predicted by the encoding end to the The brightness information is calculated on the imaging plane, and the pixel value error is calculated with the brightness image actually collected on the imaging plane. The gradient descent method is used to optimize the network parameters to realize self-supervised learning, so it is not necessary to obtain the real label of the data. In addition, this deep learning method does not require training, the network structure is designed with manual prior knowledge, and there is no need to obtain learning prior knowledge from the training set. The hologram collected in a single frame is input when used, and the parameters of the training model are reduced during the self-supervised learning process. Loss function, and finally use the output of the encoder network as the prediction of the complex amplitude. This non-end-to-end network structure also has a large number of iterative processes when solving the phase, and the time cost is high, and this method is similar to the aforementioned Yichen Wu et al. All methods only use one defocused hologram as a constraint, so the error of reconstruction results is high and the visual effect is poor.

To sum up, the existing phase recovery methods have the problems of poor interpretability, high time cost, and low image reconstruction accuracy.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a lensless holographic microscope imaging phase recovery method based on the decoupling-fusion network based on the deficiencies in the above-mentioned prior art, so as to solve the problems of poor interpretability and time in the existing phase recovery methods. High cost and low image reconstruction accuracy.

The core technical idea of the phase recovery method of the present invention is to use the multi-frame holograms collected at different defocus distances as the input of the phase recovery network, take the output of the decoupling-fusion network as the predicted complex amplitude, and combine it with the The known Fresnel diffraction model is propagated forward to each defocus plane, and the average absolute error is calculated with the brightness image actually collected by each defocus plane, and the average absolute error of each defocus plane is summed as the total loss function, and the gradient is used. The descent method updates the phase recovery network parameters. The training process of the phase recovery network is self-supervised, so it is not necessary to obtain the real complex amplitude labels of the data. The present invention saves the parameters as good initial values after pre-training on the collected or simulated small data sets, and solves the problem of actual phase recovery. When the problem occurs, it is not necessary to start a large number of gradient descent processes from random parameters, the convergence speed is faster than that of the untrained generative network, the method of the present invention requires less time, and the time cost is low. Since the convolutional neural network has a clear task in the decoupling-fusion network proposed in the present invention, it learns a simple nonlinear mapping instead of a complex inverse image reconstruction process, thus improving the generalization of the phase recovery network. performance and interpretability. In addition, the method of the present invention uses multi-frame constraints instead of single-frame constraints, which reduces the reconstruction error and improves the visual effect of amplitude or phase contrast imaging. Therefore, the reconstruction accuracy of the method of the present invention is higher, and the visual effect of the reconstructed image is better. .

Specifically, the technical scheme adopted in the present invention is as follows:

The present application provides a lensless holographic microscope imaging phase recovery method based on a decoupling-fusion network, the method includes the following steps: S1, measuring parameters of the lensless holographic microscope imaging system to be restored; S2, acquiring samples and constructing training Sample set and test sample set; S3, construct a phase recovery network; S4, train the constructed phase recovery network; S5, perform phase recovery to solve the complex amplitude.

Further, the parameters in step S1 include the center wavelength of the coherent light source, the pixels of the photoelectric sensor, and the defocus distance from the defocus plane to the sample plane.

Further, the samples in step S2 are brightness images with a resolution size of N×N of the M samples collected in the lensless holographic microscope imaging system to be restored in s defocus planes, and M groups of brightness image data are obtained, A total of M×s luminance images, where M≥300, s≥2, and N=768.

Further, the training sample set and the test sample set in step S2 are obtained by dividing the M groups of luminance image data according to a ratio of 9:1.

Furthermore, the phase recovery network in step S3 includes a decoupling network, a back propagation Fresnel diffraction layer, a fusion network, a forward propagation Fresnel diffraction layer, and a brightness extraction layer.

Further, the decoupling network includes four down-sampling modules, four up-sampling modules, and two ordinary convolution modules.

Further, the downsampling module includes three convolutional layers and one max-pooling layer for downsampling and uses the ReLU activation function, and the upsampling module includes two convolutional layers and a deconvolutional layer for upsampling. And use the ReLU activation function.

Further, the fusion network includes four down-sampling modules, four up-sampling modules, and two ordinary convolution modules.

Furthermore, the back-propagating Fresnel diffraction layer includes a discrete Fourier transform operator, a transfer function product module, and an inverse operator of the discrete Fourier transform.

Furthermore, the forward propagation Fresnel diffraction layer includes a discrete Fourier transform operator, a transfer function product module, and an inverse operator of the discrete Fourier transform.

Compared with the prior art, the beneficial effects of the present invention:

(1) The present invention uses a decoupling network to decouple dual-channel complex matrix information from a single-channel holographic brightness image, uses a fusion network to fuse multi-frame acquisition information, and combines with the Fresnel diffraction physical model, and its network learning target is clear , which is highly interpretable;

(2) The present invention uses the multi-frame luminance images collected at different defocus distances to extract information, and as a loss function constraint, the phase recovery value is closer to the true value than the single-frame recovery method, and the reconstruction accuracy rate is higher. The visual effect of the phase contrast and amplitude images is good;

(3) The phase recovery network used in the present invention is trained on a small data set, and the parameters are saved as the initial value of gradient descent. Compared with the traditional deep learning method without training, the number of gradient descent rounds required is less, and the learning Prior knowledge improves phase recovery speed.

Description of drawings

1 is a schematic diagram of a lensless holographic microscopic imaging phase recovery method based on a decoupling-fusion network provided by the present invention;

2 is a schematic diagram of a phase recovery network constructed in step S3 in the lensless holographic microscopic imaging phase recovery method based on the decoupling-fusion network provided by the present invention;

Figure 3 shows the brightness images collected from the USAF1951 resolution board at different defocus distances, in which the defocus distances corresponding to Figure 3(a), Figure 3(b), Figure 3(c), and Figure 3(d) are respectively 0.710mm, 1.185mm, 1.685mm, 2.178mm;

Figure 4 shows the brightness images of small intestinal epithelial cell sample slices collected at different defocus distances, in which the defocus distances corresponding to Figure 4(a), Figure 4(b), Figure 4(c), and Figure 4(d) are respectively 0.865mm, 1.305mm, 1.804mm, 2.304mm;

Figure 5 shows the results of phase recovery using the G-S iterative algorithm, in which Figure 5(a), Figure 5(b), Figure 5(c), and Figure 5(d) respectively show the resolution of the USAF1951 after 300 iterations of the G-S iterative algorithm. Rate plate amplitude imaging results, USAF1951 resolution plate phase contrast imaging results, small intestinal epithelial cell sample slice amplitude imaging results, small intestinal epithelial cell sample slice phase contrast imaging results;

Fig. 6 is the result of phase recovery using the deep learning algorithm proposed by Fei Wang et al., in which Fig. 6(a), Fig. 6(b), Fig. 6(c), and Fig. 6(d) are the results of using Fei Wang et al. After 300 rounds of gradient descent of the proposed deep learning algorithm, the results of the USAF1951 resolution plate amplitude imaging, the USAF1951 resolution plate phase contrast imaging results, the amplitude imaging results of the small intestinal epithelial cell sample slices, and the small intestinal epithelial cell sample slice phase contrast imaging results;

Fig. 7 is the result of phase recovery performed by the method of the present invention, wherein Fig. 7(a), Fig. 7(b), Fig. 7(c), and Fig. 7(d) are respectively the resolution of USAF1951 after 100 rounds of gradient descent by the method of the present invention. Plate amplitude imaging results, USAF1951 resolution plate phase contrast imaging results, small intestinal epithelial cell slice amplitude imaging results, and small intestinal epithelial cell sample slice phase contrast imaging results.

Detailed ways

In order to make the implementation process of the present invention clearer, a detailed description will be given below with reference to the accompanying drawings.

The present invention provides a lensless holographic microscopic imaging phase recovery method based on a decoupling-fusion network, as shown in FIG. 1 , and the specific steps are as follows:

S1, measure the parameters of the lensless holographic microscope imaging system to be restored;

Lensless holographic microscopy imaging technology is based on the principle of coaxial holography. It uses photoelectric sensors such as CMOS/CCD to be close to the sample plane, and collects the brightness image of the sample in a coherent light field or a partially coherent light field. The brightness image is the obtained hologram. Image, a technique for recovering the complete wavefront information of the sample plane using an algorithm. Specifically, the central wavelength of the coherent light source, the pixel size of the photoelectric sensor, and the defocus distance from the defocus plane to the sample plane are all parameters in the transfer function. In order to obtain the parameters in the transfer function, the central wavelength λ of the coherent light source is measured, and the pixel size p of the photoelectric sensor is obtained. The sensor in this embodiment takes a CMOS sensor as an example, and measures any s defocus planes from the sample plane at s distances from the sample plane. The focal distance {L _r |r∈1,2,…,s}, specifically, since the Fresnel diffraction occurs in the near field region, the defocus distance is 0.1mm-3mm, so that the Fresnel diffraction formula can be used to approximate Optical propagation process. The sample can be fixed, that is, the sample plane is fixed, and the photoelectric sensor can be moved, that is, the defocused plane can be moved, or the photoelectric sensor can be fixed, and the sample can be moved; Stablize.

S2, obtain samples and construct training sample sets and test sample sets;

The samples in this embodiment are brightness images of M samples collected in a lensless holographic microscope imaging system with a resolution size of N×N in s defocus planes, and M groups of brightness image data are obtained, a total of M×s sheets Brightness image, where M≥300, which can ensure the sufficiency of the training data set; s≥2, at least two frames, to ensure that multi-frame reconstruction can be performed; N=768, which can adapt to the memory size of this embodiment. This example uses bmp format pictures, and the network structure is large. For a graphics card with 12GB video memory, it is more suitable to use an image with a resolution of 768×768. For other graphics card devices with smaller or larger video memory, the value of N needs to be Adjustment; in addition, the parameters trained on the training sample set of 768×768 resolution can also complete the restoration work for input of other resolutions. When implementing, use the image input of 512×512 or 1024×1024 resolution. The phase recovery network can be recovered, because the convolution operation of the neural network is a 3×3 convolution kernel sliding operation, so there is no mandatory requirement for the input resolution. Randomly, the M groups of brightness image data are divided into training sample sets and test sample sets according to the ratio of 9:1, that is, 9M/10 groups of brightness image data in the M groups of brightness image data form the training sample set, and the remaining 1M/10 groups of brightness images. The image data constitutes the test sample set.

S3, construct a phase recovery network;

As shown in Figure 2, the present invention uses a phase recovery network to replace the nonlinear mapping part in the inverse problem of image reconstruction. The phase recovery network uses multiple frames of luminance images collected at different defocus distances as input, and the decoupling network is used to convert the The single-channel luminance image collected in each frame is mapped to a two-channel complex matrix and back-propagated to the sample plane through the Fresnel diffraction model. The information is fused by multiple complex number matrices through the fusion network, and the predicted complex amplitude is output. The loss function side forwards the predicted complex amplitude to the acquisition plane at each defocus distance through the known Fresnel diffraction model, calculates the average absolute error with the brightness image actually collected on the corresponding plane, and propagates back through the gradient. , using gradient descent to optimize the parameters of the phase recovery network. In Figure 2, there are s brightness extraction layers, reconstruction loss, decoupling network, back propagation Fresnel diffraction layer, and forward propagation Fresnel diffraction layers, which correspond to samples collected at s defocus distances, respectively. Among them, the s reconstruction losses are combined into a total loss function. As the training progresses, the parameters of the s decoupling networks are different due to the different mapping relationships.

S31, constructing a phase recovery network;

As shown in Figure 2, the phase recovery network includes a decoupling network, a back propagation Fresnel diffraction layer, a fusion network, a forward propagation Fresnel diffraction layer, and a brightness extraction layer. Both the decoupling network and the fusion network adopt the common U-shaped network structure, but the number of channels in each convolutional layer is different.

其中第一个通道张量为实部矩阵，第二个通道张量为虚部矩阵，而光波本身为一个复数矩阵，这样更加能够反映实际的成像过程。具体地，每个下采样模块中包括三个卷积层和一个用作下采样的最大池化层并使用ReLU函数作为前两个卷积层的激活函数，卷积层用于提取特征，使得通道数逐渐增加，其中池化操作会降低分辨率。每个上采样模块中包括一个用作上采样的反卷积层和两个卷积层并使用ReLU激活函数，其中反卷积层用于增加分辨率。使用残差连接将下采样模块中第二个卷积层的输出张量和第三个卷积层输出的同等大小张量相加，残差结构网络容易优化，能够缓解深度神经网络增加深度带来的梯度消失的问题，同时也能够提升解耦网络的训练速度，降低时间成本；采用跳层连接将下采样模块最大池化层输出的张量和对应上采样模块中反卷积层输出的同等大小张量沿通道方向连接，作为该上采样模块中第一个卷积层的输入，这样能够将较浅的卷积层特征引出，保留丰富的低层级信息，弥补由于池化操作带来的低层级图像信息丢失和分辨率降低，从而提高图像重构的准确率，同时还能够减少梯度消失和网络退化问题。下采样模块、上采样模块、普通卷积模块相关的具体参数设置详见表1。更具体地，解耦网络的设置依次为：第一下采样模块→第二下采样模块→第三下采样模块→第四下采样模块→第一普通卷积模块→第一上采样模块→第二上采样模块→第三上采样模块→第四上采样模块→第二普通卷积模块，其中第一普通卷积模块用于连接解耦网络上采样和下采样的路径，第二普通卷积模块用于使解耦网络输出所需大小的张量信息。The decoupling network includes four down-sampling modules, four up-sampling modules, and two ordinary convolution modules, which are used to convert the out-of-focus plane brightness image brightness matrix {I _r |r∈1,2,…, s} decoupled into a complex matrix of two-channel tensors

The first channel tensor is the real part matrix, the second channel tensor is the imaginary part matrix, and the light wave itself is a complex number matrix, which can better reflect the actual imaging process. Specifically, each downsampling module includes three convolutional layers and a max pooling layer used for downsampling and uses the ReLU function as the activation function of the first two convolutional layers. The convolutional layers are used to extract features such that The number of channels is gradually increased, where the pooling operation reduces the resolution. Each upsampling module includes a deconvolution layer for upsampling and two convolution layers and uses the ReLU activation function, where the deconvolution layer is used to increase the resolution. Using residual connection to add the output tensor of the second convolutional layer in the downsampling module and the same size tensor output by the third convolutional layer, the residual structure network is easy to optimize, which can alleviate the increase of the depth band of the deep neural network. At the same time, it can also improve the training speed of the decoupling network and reduce the time cost; the jump layer connection is used to connect the tensor output by the maximum pooling layer of the downsampling module and the output tensor of the deconvolution layer in the corresponding upsampling module. Tensors of the same size are connected along the channel direction as the input of the first convolutional layer in the upsampling module, which can extract the shallower convolutional layer features, retain rich low-level information, and make up for the low level caused by the pooling operation. Hierarchical image information is lost and resolution is reduced, thereby improving the accuracy of image reconstruction, while reducing gradient disappearance and network degradation. The specific parameter settings related to the down-sampling module, the up-sampling module and the common convolution module are shown in Table 1. More specifically, the setting of the decoupling network is as follows: first downsampling module→second downsampling module→third downsampling module→fourth downsampling module→first ordinary convolution module→first upsampling module→th The second upsampling module → the third upsampling module → the fourth upsampling module → the second ordinary convolution module, where the first ordinary convolution module is used to connect the path of decoupling network upsampling and downsampling, and the second ordinary convolution module The module is used to make the decoupling network output tensor information of the desired size.

由离焦平面反向传播到样本平面，反向传播菲涅尔衍射的线性运算记为{F^-1 _r|r∈1,2,…,s}：The back-propagating Fresnel diffraction layer includes discrete Fourier transform operators, transfer function product modules, and inverse operators of discrete Fourier transforms. complex matrix used to output the s decoupling networks

Backpropagating from the out-of-focus plane to the sample plane, the linear operation of backpropagating Fresnel diffraction is denoted as {F ^-1 _r |r∈1,2,…,s}:

Among them, F represents the discrete Fourier transform operator; F ^-1 represents the inverse operator of the discrete Fourier transform; f _x and f _y are frequency domain units, and H _r (f _x , f _y ) represents the transfer function matrix , the expression is:

Among them, j is the imaginary unit; L _r is the defocusing distance from the rth defocusing plane to the sample plane; λ is the center wavelength of the light source;

具体地，输入层将反向传播菲涅尔衍射层输出的s个复数矩阵求和，每个下采样模块中包括三个卷积层和一个用作下采样的最大池化层并使用ReLU激活函数，卷积层用于提取特征，使得通道数逐渐增加，其中池化操作会降低分辨率。每个上采样模块中包括两个卷积层和一个用作上采样的反卷积层并使用ReLU激活函数，其中反卷积层用于增加分辨率，具体参数设置如表1所示。使用残差连接将下采样模块中第二个卷积层的输出张量和第三个卷积层输出的同等大小张量相加，残差结构网络容易优化，能够缓解深度神经网络增加深度带来的梯度消失的问题，同时也能够提升解耦网络的训练速度，降低时间成本；采用跳层连接将下采样模块最大池化层输出的张量和对应上采样模块中反卷积层输出的同等大小张量沿通道方向连接，作为该上采样模块中第一个卷积层的输入，这样能够将较浅的卷积层特征引出，保留丰富的低层级信息，弥补由于池化操作带来的低层级图像信息丢失和分辨率降低，从而提高图像重构的准确率，同时还能够减少梯度消失和网络退化问题。下采样模块、上采样模块、普通卷积模块相关的具体参数设置详见表1。更具体地，融合网络的设置依次为：第五下采样模块→第六下采样模块→第七下采样模块→第八下采样模块→第三普通卷积模块→第五上采样模块→第六上采样模块→第七上采样模块→第八上采样模块→第四普通卷积模块，其中第三普通卷积模块用于连接融合网络上采样和下采样的路径，第四普通卷积模块用于使融合网络输出所需大小的张量信息。The fusion network includes four down-sampling modules, four up-sampling modules, and two ordinary convolution modules, which are used to fuse the information of the hologram luminance images collected in multiple frames and generate the complex amplitude predicted by the sample plane.

Specifically, the input layer sums the s complex matrices output by the back-propagated Fresnel diffraction layer, and each downsampling module includes three convolutional layers and a max pooling layer used as downsampling and uses ReLU activation function, the convolutional layer is used to extract features, so that the number of channels is gradually increased, and the pooling operation will reduce the resolution. Each upsampling module includes two convolutional layers and a deconvolutional layer for upsampling and uses the ReLU activation function, where the deconvolutional layer is used to increase the resolution, and the specific parameter settings are shown in Table 1. Using residual connection to add the output tensor of the second convolutional layer in the downsampling module and the same size tensor output by the third convolutional layer, the residual structure network is easy to optimize, which can alleviate the increase of the depth band of the deep neural network. At the same time, it can also improve the training speed of the decoupling network and reduce the time cost; the jump layer connection is used to connect the tensor output by the maximum pooling layer of the downsampling module and the output tensor of the deconvolution layer in the corresponding upsampling module. Tensors of the same size are connected along the channel direction as the input of the first convolutional layer in the upsampling module, which can extract the shallower convolutional layer features, retain rich low-level information, and make up for the low level caused by the pooling operation. Hierarchical image information is lost and resolution is reduced, thereby improving the accuracy of image reconstruction, while reducing gradient disappearance and network degradation. The specific parameter settings related to the down-sampling module, the up-sampling module and the common convolution module are shown in Table 1. More specifically, the settings of the fusion network are in the following order: the fifth downsampling module→the sixth downsampling module→the seventh downsampling module→the eighth downsampling module→the third ordinary convolution module→the fifth upsampling module→the sixth Upsampling module→seventh upsampling module→eighth upsampling module→fourth ordinary convolution module, where the third ordinary convolution module is used to connect the path of fusion network upsampling and downsampling, and the fourth ordinary convolution module is used for It is used to make the fusion network output tensor information of the required size.

由样本平面正向传播到s个离焦平面，输出s个离焦平面的复振幅预测矩阵

The forward propagation Fresnel diffraction layer includes a discrete Fourier transform operator, a transfer function product module, and an inverse operator of the discrete Fourier transform. The predicted complex amplitude used to fuse the network output

Forward propagation from the sample plane to s out-of-focus planes, and output the complex amplitude prediction matrix of s out-of-focus planes

为融合网络在样本平面预测的复振幅。Among them, F represents the discrete Fourier transform operator; F ^-1 represents the inverse operator of the discrete Fourier transform; H _r (f _x , f _y ) represents the transfer function matrix, and f _x and f _y are frequency domain units ,

Complex amplitude predicted at the sample plane for the fusion network.

中提取出亮度图像矩阵

这样方便与光电传感器探测到的实际亮度图像进行对比，求出损失。提取亮度信息的表达式为：The luminance extraction layer includes a computational module for the s out-of-focus plane complex amplitude prediction matrices output from the forward-propagating Fresnel diffraction layer

The luminance image matrix is extracted from

In this way, it is convenient to compare with the actual brightness image detected by the photoelectric sensor to find the loss. The expression for extracting luminance information is:

代表

张量的第一个通道，代表复振幅的实部；

代表

张量的第二个通道，代表复振幅的虚部。in,

represent

The first channel of the tensor, representing the real part of the complex amplitude;

represent

The second channel of the tensor, representing the imaginary part of the complex amplitude.

The invention uses the decoupling network to decouple the dual-channel complex matrix information from the single-channel holographic brightness image, uses the fusion network to fuse the multi-frame acquisition information, and combines it with the Fresnel diffraction physical model. The network learning target is clear and interpretable. powerful.

Table 1: Parameter settings of decoupling network and fusion network in phase recovery network.

S32, define the total loss function L of the phase recovery network.

The expression of the total loss function L of the phase recovery network is as follows:

为亮度提取层输出的第 r个离焦平面预测的全息图亮度矩阵，I_r为第r个离焦平面实际采集到的全息图亮度矩阵。采用1-范数，对细节信息友好，能够有效提高分辨率。由总损失函数L的表达式能够得到采集到的亮度和根据预测的复振幅通过亮度提取得到的亮度之间的差值，差值越小，说明预测的复振幅越准确，差值越大，说明预测的复振幅越不准确，从而能够反映出本发明方法相位恢复的准确度。Among them, mean(*) is the pixel-by-pixel mean operator,

is the hologram brightness matrix predicted by the rth defocusing plane output by the brightness extraction layer, and I _r is the hologram brightness matrix actually collected by the rth defocusing plane. The 1-norm is adopted, which is friendly to detail information and can effectively improve the resolution. From the expression of the total loss function L, the difference between the collected luminance and the luminance obtained by luminance extraction according to the predicted complex amplitude can be obtained. The smaller the difference is, the more accurate the predicted complex amplitude is, and the larger the difference is. It is explained that the more inaccurate the predicted complex amplitude is, so that the accuracy of the phase recovery of the method of the present invention can be reflected.

The present invention uses the multi-frame holograms collected under different defocus distances to extract information, and as a loss function constraint, the phase recovery value is closer to the true value than the single-frame recovery method, the reconstruction accuracy rate is higher, and the reconstruction phase difference and The visual effect of the amplitude image is good.

S4, train the constructed phase recovery network;

The brightness images {I _r |r∈1,2,…,s} of the 9M/10 groups in the training sample set collected in s defocus planes are used as the input of the phase recovery network of the present invention, and the phase recovery network is performed J times. For iterative training, J is greater than 100, so as to ensure that the network is fully trained, specifically, 300 times in this embodiment. Each round needs to traverse all training sample sets and test sample sets. First, all samples in the training sample set are input into the phase recovery network in turn to train the network and optimize network parameters; then all samples in the test sample set are input into the phase recovery network in turn. , which is used to test whether the trained network needs to continue training. After the test sample set reaches a certain loss function requirement, the parameters of the phase recovery network are saved; in this embodiment, the Adam optimizer provided by the PyTorch framework is used to optimize the parameters in the phase recovery network, and the learning rate is set to 0.0001. The average loss on the sample set is less than or equal to 3, which can ensure the accuracy of the prediction. The average loss refers to the average of the total loss L of all test samples.

The phase recovery network used in the present invention only needs to be trained on a small data set and does not require a large amount of data, which can effectively reduce the training time, reduce the time cost, and save the parameters as the initial value of gradient descent. The deep learning method for training requires fewer gradient descent rounds and improves the speed of phase recovery by learning prior knowledge.

S5, perform phase recovery to solve the complex amplitude;

即为所求复振幅，即求得复振幅完成了相位恢复。The parameters saved in step S4 are loaded into the phase recovery network, and under the same system parameters as the phase recovery network training, the brightness image data of the tested sample in s defocused planes are collected, and input into the phase recovery network. Calculate the total loss function, and use the Adam optimizer to perform the gradient descent optimization parameter K rounds again until the total loss function on the input data satisfies L≤α, where α is the manually selected loss function threshold. Specifically, the loss function threshold α is equal to 0.5, to ensure the phase recovery accuracy of the phase recovery network, and fuse the prediction results of the sample plane complex amplitude output by the network

That is, the complex amplitude is obtained, that is, the complex amplitude is obtained to complete the phase recovery.

Implementation conditions and result analysis of the embodiment of the present invention:

1. Implementation conditions;

The method of the present invention is applicable to any mutually compatible hardware and software platforms. The hardware test platform adopted in this embodiment is: Intel Core i7 CPU with a main frequency of 3.60GHz and 16GB of memory; GPU: NVIDIA TITAN XP, 12GB of video memory; software The simulation platform is: Windows 10 64-bit operating system; software simulation language: Python; using deep learning framework: PyTorch.

2. Result analysis.

The present invention, the G-S iterative algorithm and the method of Fei Wang et al. are used to perform phase recovery on the experimental data collected by the same system. The collected images are shown in Figure 3, and the phase contrast and amplitude imaging results are shown in Figure 4.

Figure 3 shows the brightness images collected from the USAF1951 resolution board at different defocus distances, in which the defocus distances corresponding to Figure 3(a), Figure 3(b), Figure 3(c), and Figure 3(d) are respectively 0.710mm, 1.185mm, 1.685mm, 2.178mm. The present invention uses multiple frames of luminance images collected at different defocus distances to extract information, and as a loss function constraint, the phase recovery value is closer to the real value than the single-frame recovery method, the reconstruction accuracy rate is higher, and the reconstruction phase difference and The visual effect of the amplitude image is good.

Figure 4 shows the brightness images of small intestinal epithelial cell sample slices collected at different defocus distances, in which the defocus distances corresponding to Figure 4(a), Figure 4(b), Figure 4(c), and Figure 4(d) are respectively 0.865mm, 1.305mm, 1.804mm, 2.304mm. The present invention uses multiple frames of luminance images collected at different defocus distances to extract information, and as a loss function constraint, the phase recovery value is closer to the real value than the single-frame recovery method, the reconstruction accuracy rate is higher, and the reconstruction phase difference and The visual effect of the amplitude image is good.

Figure 5 shows the results of phase recovery using the G-S iterative algorithm, in which Figure 5(a), Figure 5(b), Figure 5(c), and Figure 5(d) are the results of using the G-S iterative algorithm for 300 rounds of gradient descent on the USAF1951 Resolution plate amplitude imaging results, USAF1951 resolution plate phase contrast imaging results, small intestinal epithelial cell sample slice amplitude imaging results, small intestinal epithelial cell sample slice phase contrast imaging results. Specifically, the results of FIG. 5 were obtained with the method disclosed by Gerchberg and Saxton in an article entitled "A practical algorithm for the determination of the phase from image and diffraction plane pictures" in 1972.

Fig. 6 is the result of phase recovery using the deep learning algorithm proposed by Fei Wang et al., in which Fig. 6(a), Fig. 6(b), Fig. 6(c), and Fig. 6(d) are the results of using Fei Wang et al. After 300 rounds of gradient descent by the proposed deep learning algorithm, the results of the USAF1951 resolution plate amplitude imaging, the USAF1951 resolution plate phase contrast imaging results, the amplitude imaging results of small intestinal epithelial cell samples, and the phase contrast imaging results of small intestinal epithelial cell samples were obtained. Specifically, the results in Figure 6 are obtained by the method disclosed in an article titled "Phase imaging with anuntrained neural network" published by Fei Wang et al. in the journal "Light: Science & Applications" in 2020.

Fig. 7 is the result of phase recovery performed by the method of the present invention, wherein Fig. 7(a), Fig. 7(b), Fig. 7(c), and Fig. 7(d) are respectively the resolution of USAF1951 after 100 rounds of gradient descent by the method of the present invention. Plate amplitude imaging results, USAF1951 resolution plate phase contrast imaging results, small intestinal epithelial cell slice amplitude imaging results, and small intestinal epithelial cell sample slice phase contrast imaging results. On the one hand, the method of the present invention is trained on a small data set, and the parameters are saved as the initial value of gradient descent. Compared with the traditional deep learning method without training, the number of gradient descent rounds required is less, and the number of iteration rounds of the method of the present invention is 100. Less than 300 for the two comparison methods, the phase recovery speed of the method of the present invention is faster. On the other hand, by comparing Figure 5, Figure 6, and Figure 7, it can be seen that in the phase recovery results of the G-S iterative algorithm and the existing deep learning method, there are obvious noises and shadows for the sample amplitude imaging, indicating that the G-S iterative algorithm will be more efficient. Quickly fall into the local optimal solution, resulting in the phase recovery value is not smooth enough. The method of the present invention learns the prior knowledge of a large amount of data, so the image is smoother, with less noise and less shadow; the existing deep learning method only uses a single image, so Insufficient constraints result in inaccurate phase recovery values. The method of the present invention uses multiple frames and has many constraints, resulting in better recovery effects and more accurate values. The amplitude recovery result obtained by the method of the present invention has almost no noise and shadow; for sample phase contrast imaging, there are different degrees of overexposure, and the phase recovery result obtained by the method of the present invention has no obvious overexposure phenomenon and has better visual effect. Because the present invention uses the multi-frame brightness images collected at different defocus distances to extract information, and as a loss function constraint, the phase recovery value is closer to the real value than the single-frame recovery method, the reconstruction accuracy is higher, and the reconstruction phase difference And the visual effect of the amplitude image is good.

The above are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims (10)

1. a lensless holographic microscopic imaging phase recovery method based on decoupling-fusion network, is characterized in that, described method comprises the steps: S1, measure the parameters of the lensless holographic microscope imaging system to be restored; S2, obtain samples and construct training sample sets and test sample sets; S3, construct a phase recovery network; S4, train the constructed phase recovery network; S5, perform phase recovery to solve the complex amplitude. 2 . The lensless holographic microscope imaging phase recovery method based on decoupling-fusion network according to claim 1 , wherein the parameters in the step S1 include the center wavelength of the coherent light source, the pixel of the photoelectric sensor. 3 . , the defocus distance from the defocus plane to the sample plane. 3. The lensless holographic microscopic imaging phase recovery method based on the decoupling-fusion network according to claim 2, wherein the sample in the step S2 is the lensless holographic microscopic imaging to be recovered The M samples collected in the system have brightness images with a resolution size of N×N in s defocus planes, and M groups of brightness image data are obtained, a total of M×s brightness images, where M≥300, s≥2, N = 768. 4. The lensless holographic microscope imaging phase recovery method based on decoupling-fusion network according to claim 3, wherein the training sample set and the test sample set in the step S2 are The M groups of luminance image data are divided and obtained according to the ratio of 9:1. 5. The lensless holographic microscopic imaging phase recovery method based on the decoupling-fusion network according to claim 4, wherein the phase recovery network in the step S3 comprises a decoupling network, a back-propagating phenanthrene Nell diffraction layer, fusion network, forward propagation Fresnel diffraction layer, brightness extraction layer. 6 . The lensless holographic microscope imaging phase recovery method based on decoupling-fusion network according to claim 5 , wherein the decoupling network comprises four down-sampling modules, four up-sampling modules, two Ordinary convolution module. 7. The lensless holographic microscopy imaging phase recovery method based on decoupling-fusion network according to claim 6, wherein the downsampling module comprises three convolutional layers and a max pool used for downsampling The upsampling module includes two convolutional layers and a deconvolutional layer for upsampling and uses the ReLU activation function. 8. The lensless holographic microscope imaging phase recovery method based on decoupling-fusion network according to claim 7, wherein the fusion network comprises four down-sampling modules, four up-sampling modules, two common convolution module. 9 . The lensless holographic microscopic imaging phase recovery method based on the decoupling-fusion network according to claim 8 , wherein the back-propagating Fresnel diffraction layer comprises a discrete Fourier transform operator, a transfer Function product module, inverse operator of discrete Fourier transform. 10 . The lensless holographic microscopic imaging phase recovery method based on decoupling-fusion network according to claim 9 , wherein the forward propagation Fresnel diffraction layer comprises a discrete Fourier transform operator, a transfer Function product module, inverse operator of discrete Fourier transform.

Images