CN114387395A - A fast generation method of hologram based on phase-dual resolution network - Google Patents

Abstract

The invention discloses a method for rapidly generating a hologram based on a phase-double-resolution network. The method comprises the following steps: without making a real pure-phase hologram mask, utilizing the differentiability of an angular spectrum method, and using a natural image to realize convolution Unsupervised training of neural networks; convolutional layers learn feature maps in the same space instead of across distances; use atrous convolution and group convolution to reduce network model computation and GPU memory usage; use MS‑SSIM loss and the MSE loss are combined as a consistency loss function to generate reconstruction maps that are more in line with the human visual system. This application calculates and generates a hologram with a resolution of 1080P in only 57 milliseconds, and the peak signal-to-noise ratio between the optimal numerical reconstruction image and the target intensity image reaches 31.17dB.

Description

A fast generation method of hologram based on phase-dual resolution network

technical field

The invention relates to the field of computationally generated holography and computer vision, in particular to a method for rapidly generating a hologram based on a phase-dual resolution network.

Background technique

Holographic display is considered to be an advanced display technology, and is gradually applied in 3D scene reconstruction, virtual reality and augmented display systems, enabling a 3D display mode without glasses. While realizing the real-time holographic display, how to ensure the high fidelity of the holographic display image is still a difficult problem. The phase-type spatial light modulator has high optical efficiency, and there is no interference of conjugate images in the reconstruction process. Pure-phase holograms have become the main coding method for computationally generated holograms. Therefore, it is of great significance for the development of holographic display technology to develop an algorithm that can calculate and generate high-quality phase holograms in real time.

For more than half a century, a variety of algorithms for computationally generating holograms have been introduced, mainly including iterative optimization algorithms and non-iterative optimization algorithms. The Gercheberg-Saxton algorithm proposed in 1972 is a typical representative of iterative optimization to generate holograms. Then there are some improved algorithms based on the Gercheberg-Saxton iterative optimization algorithm, such as adding random noise to the Gercheberg-Saxton iterative optimization algorithm to generate holograms; bidirectional error diffusion algorithm to generate holograms; gradient descent or Wirtinger derivatives to solve non-convex optimization The problem is indirect computationally generating holograms, etc. Since the iterative optimization algorithm takes a long time to generate holograms, it is not suitable for the real-time calculation of large-scale and high-resolution holograms. Therefore, some researchers have proposed non-iterative optimization algorithms, such as two-phase amplitude encoding and one-step phase encoding. Extraction algorithms, etc. Although the calculation speed of the non-iterative optimization algorithm has been greatly improved and can meet the real-time calculation, the reconstructed image of the hologram contains more speckle noise. Therefore, non-iterative optimization algorithms cannot guarantee the quality of computationally generated holograms.

In recent years, with the deepening of algorithm research, deep learning technology has been gradually introduced into the field of optics, and the emergence of deep learning-related algorithms has also provided a new method for computationally generating holograms. Convolutional neural networks act as generalized approximation functions that learn the mapping between input and output. In 2021, Shi et al. introduced a large-scale Fresnel holography dataset to train a convolutional neural network to generate realistic 3D holograms with the ability to change the focal length. Some researchers also use the Fresnel method to generate a large number of holograms as a dataset to train a confrontational generative network. The training sets used in these methods are all holograms generated by traditional iterative optimization algorithms. Therefore, the performance of the network model is directly limited by the quality of the training set, and indirectly limited by the traditional iterative optimization algorithm. How to break these limitations becomes a key challenge for the application of deep learning to computationally generated holograms. More importantly, convolutional layers act on the spatial dimension of the input and are best suited for modeling and computing the spatial relationship mapping between input and output. The convolutional neural network maps the image defined in the object plane to the pure phase hologram mask defined in the hologram plane. There is a spatial correspondence that cannot be preserved, and the powerful feature mapping capability of the convolutional neural network cannot be exerted. The problem.

SUMMARY OF THE INVENTION

In order to solve the above-mentioned problems in the background technology, the present application provides a method for rapidly generating a hologram based on a phase-dual resolution network. The main technical solutions are as follows:

Build phase-dual resolution network models f _net1 and f _net2 ;

The trained phase-dual-resolution network f _net1 calculates the initial phase φ ₀ according to the input target intensity map I;

Calculate the complex-valued wave field U _z according to the initial phase φ ₀ and the target intensity map I; calculate the complex-valued wave field U ₀ obtained after the complex-valued wave field U _z propagates in free space -z according to the angular spectrum method;

The trained phase-dual-resolution network _fnet2 computes a phase-only hologram from the input complex-valued wavefield U ₀ .

Preferably, the building a phase-dual resolution network model includes:

The phase-dual resolution network is mainly divided into two parts: encoder and decoder;

In the encoder of the phase-dual resolution network, the first two layers of the network use group convolution for feature extraction; the remaining network layers in the encoder use atrous convolution for feature extraction, where the atrous factor in the atrous convolution follows the The formula for calculating the maximum distance between two non-zero values in convolution:

M _i =max[M _i+ 1-2r _i ,M _i+ 1-2(M _i+1 -r _i ),r _i ],M _n =r _n ;

In the decoder, the features extracted by the encoder are upsampled by sub-pixel convolution until it is the same resolution as the input image; the third layer network output of the encoder is connected to the first layer of the decoder for feature stitching using skip connections ;Use 1×1 convolution as the last layer of the decoder, so that the output is a single-channel image;

Normalize the grayscale values of a single-channel image to [-π,π].

Preferably, the training process of the trained phase-dual resolution network includes:

其中重建距离为z；The constructed phase-dual-resolution network f _net2 is calculated to generate a pure phase hologram, and the reconstructed image of the hologram is calculated according to the angular spectrum method

where the reconstruction distance is z;

之间的MS-SSIM损失和MSE损失；Calculate the input target intensity map I and the reconstructed map calculated by the angular spectrum method

between MS-SSIM loss and MSE loss;

The MS-SSIM loss and the MSE loss are summed according to a certain weight ratio as the overall consistency loss function to realize the simultaneous training of the phase-dual-resolution network models f _net1 and f _net2 .

Preferably, the calculation formula of the complex-valued wave field U _z includes:

Preferably, the MS-SSIM loss and the MSE loss are summed according to a certain weight ratio as an overall loss function including:

表示重建图，

用于计算I和

之间的多尺度结构相似性，p表示像素，用于逐像素计算误差，依据经验设置α＝0.84。where I represents the target intensity map,

represents the reconstructed graph,

used to calculate I and

The multi-scale structural similarity between , p represents the pixel, and is used to calculate the error pixel by pixel, and α = 0.84 is set empirically.

Description of drawings

Fig. 1 shows the overall structure diagram of the pure phase hologram generated based on the phase-dual resolution network of the present application;

Fig. 2 shows the phase-dual resolution network structure diagram of the present application;

FIG. 3 shows a schematic diagram of the holographic display structure and a simulated physical diagram of the holographic display constructed in the present application.

Detailed ways

The present invention will be clearly and completely described below with reference to the accompanying drawings and in conjunction with the preferred embodiments.

Holographic display technology is gradually applied in 3D scene reconstruction, virtual reality and augmented display systems, making 3D display mode without glasses possible. In the holographic display technology, the hologram, as the carrier of information, contains the depth and intensity information of the object, so the computationally generated hologram is a key step in the holographic display. The current methods for generating holograms by computation mainly include iterative optimization algorithms and non-iterative optimization algorithms. However, both algorithms have traded off optimal quality of holograms and computation time for many years. With the gradual introduction of deep learning technology into the field of optics research, many satisfactory results have been achieved. The invention of a neural network algorithm capable of generating high-quality holograms by real-time calculation is of great significance to the development of real-time holographic display technology.

To this end, the present application provides a method for fast generation of holograms based on a phase-dual resolution network. First, the angular spectrum method of light field propagation in free space is introduced, and its calculation formula is as follows:

u _z (x ₁ , y ₁ )=f _AS {φ(x,y)}=IFFT{FFT{u _src (x,y)·e ^iφ(x,y) }·H(f _x ,f _y ) } (1)

where u _src represents the complex wave field generated by the coherent light source, u _z represents the complex valued wave field at the diffraction distance z, f(x, y) represents the pure phase hologram loaded into the spatial light modulator, FFT and IFFT are respectively Fourier transform and inverse Fourier transform operators, H(f _x , f _y ) is the angular spectral transform function. The expression of H(f _x , f _y ) is expressed as:

where f _x and f _y are the spatial frequencies; λ is the wavelength of light, z is the diffraction distance, and H(f _x , f _y ) is the transfer function of angular spectroscopy. It can be known from equation (1) that u _src (x, y) is a content-independent coherent light source, so the derivability of angular spectroscopy is only related to H(f _x , f _y ). According to equation (2), the transfer function of angular spectroscopy is related to wavelength, propagation distance, pixel pitch and spatial resolution. When the optical system is fixed, the four parameters are fixed, and the transfer function can be regarded as a constant in the backpropagation of the convolutional neural network. Therefore, the calculation formula of the angular spectrum analysis method satisfies the gradient propagation. According to the general approximation theorem of neural network, the pure phase hologram is generated by forward propagation of convolutional neural network, and the numerical reconstruction of the hologram is calculated by the angular spectrum method. This process can be expressed in the following form:

表示再现图像。通过损失函数和角谱法计算梯度的过程可以表示为：where I represents the target amplitude, _fnet represents the approximate function of the convolutional neural network, φ _holo represents the pure phase hologram,

Indicates the reproduced image. The process of calculating the gradient through the loss function and the angle spectrum method can be expressed as:

where w _k and b _k represent the kth layer trainable parameters of the convolutional neural network, and φ _holo represents the hologram.

Figure 1 shows the overall structure of a fast generation method for holograms based on a phase-dual resolution network. The input is any intensity map, and the trained phase-dual-resolution network f _net1 maps it to a phase map; the phase map and the input target intensity map are calculated as a complex-valued wave field U by formula (5). _z ; According to the angular spectrum method, the complex-valued wavefield U ₀ is obtained after the complex-valued wavefield propagates in the free space -z distance; since the phase-dual resolution network f _net2 cannot perform complex number operations, the complex-valued wavefield is calculated according to Euler's formula. The wave field U ₀ is decomposed into real and imaginary parts, and the matrix splicing operation is performed; the eigenmatrix after matrix splicing is input into the phase-dual resolution network f _{net2 to} calculate the hologram.

此时式 (3)可进一步表示为：The above process is the process of generating holograms using the trained phase-dual resolution networks f _net1 and f _net2 . Since high-quality holographic masks are difficult to obtain, the training process is done by convolutional neural network training without holographic masks. In the training phase, the hologram calculated by the phase-double-resolution network f _net2 is calculated by the angular spectrum method.

At this time, formula (3) can be further expressed as:

表示再现图像，φ_holo表示纯相位全息图，f_AS表示角谱法，I表示目标振幅。分别计算输入的目标强度图I和角谱法计算的重建图

之间的MS-SSIM损失和MSE损失，并将其按照一定的比例求和作为整体的一致性损失函数，实现相位-双分辨率网络模型f_net1和f_net2的同时训练。一致性损失函数可表示为：In the formula,

represents the reproduced image, φ _holo represents the pure phase hologram, f _AS represents the angular spectrum method, and I represents the target amplitude. Calculate the input target intensity map I and the reconstructed map calculated by the angular spectrum method respectively

The MS-SSIM loss and the MSE loss between them are summed according to a certain ratio as the overall consistency loss function to realize the simultaneous training of the phase-dual-resolution network models f _net1 and f _net2 . The consistency loss function can be expressed as:

表示重建图，

用于计算I和

之间的多尺度结构相似性，p表示像素，用于逐像素计算误差，依据经验设置α＝0.84。where I is the target amplitude,

represents the reconstructed graph,

used to calculate I and

The multi-scale structural similarity between , p represents the pixel, and is used to calculate the error pixel by pixel, and α = 0.84 is set empirically.

Figure 2 shows a phase-dual resolution network structure diagram. where CBP represents the convolution module, which includes a convolution layer, a batch normalization layer, and a PReLU activation function. RB stands for residual block, and RBB stands for residual bottleneck block. DAPPM stands for Deep Aggregation Pyramid Pooling Module. The phase-dual resolution network is mainly composed of an encoder and a decoder, where the encoder is composed of a high-resolution branch and a low-resolution branch.

The high-resolution branch of the encoder can retain more high-frequency features, while the low-resolution branch can extract high-level abstract features of the target amplitude. The first two layers of the encoder network use group convolutions for feature extraction. The remaining network layers in the encoder use atrous convolution for feature extraction, where the atrous factor in atrous convolution follows the formula for calculating the maximum distance between two non-zero values in the convolution:

M _i =max[M _i+1 -2r _i ,M _i+1 -2(M _i+1 -r _i ), _{r i} ],M _n =rn (8)

where i represents the i-th layer convolution, and M _i represents the maximum distance between non-zero values in the i-th layer convolution output. r _i represents the dilation factor of the i-th layer of the convolution. The use of group convolution can reduce the amount of computation and GPU memory usage when generating high-resolution holograms by the phase-dual resolution network, and the use of atrous convolution can expand the receptive field of convolution operations to enhance feature extraction capabilities.

In the decoder, the features extracted by the encoder are upsampled by sub-pixel convolutions until the same resolution as the input image. The encoder's third-layer network output uses skip connections to the decoder's first layer for feature concatenation, enhancing feature decoding capabilities. A 1×1 convolution is used as the last layer of the decoder so that the output is a single-channel image. The grayscale values of the single-channel images are normalized to [-π,π] so that the final output of the network is a phase map.

FIG. 3 shows a schematic diagram of a holographic display structure and a simulated physical diagram of the holographic display constructed in the present application. The model of the spatial light modulator is Holoeye Photonics AG, the pixel pitch is 8 μm, the laser wavelength is 532 nm, and the propagation distance is 40 cm.

Claims (5)

1. a kind of hologram fast generation method based on phase-double resolution network, it is characterized in that: (1) Build phase-dual resolution network models f _net1 and f _net2 ; (2) The trained phase-dual resolution network f _net1 calculates the initial phase φ ₀ according to the input target intensity map I; (3) Calculate the complex-valued wave field U _z according to the initial phase φ ₀ and the target intensity map I; according to the angular spectrum method, calculate the complex-valued wave field U ₀ obtained after the complex-valued wave field U _z propagates in free space -z ; (4) The trained phase-dual-resolution network f _net2 calculates a phase-only hologram according to the input complex-valued wave field U ₀ . 2. The method according to claim 1, wherein building a phase-dual resolution network model comprises: (1) In the encoder of the phase-dual resolution network, the first two layers of the network use group convolution for feature extraction; the remaining network layers in the encoder use atrous convolution for feature extraction, and the holes in atrous convolution are used for feature extraction. factor, following the formula for calculating the maximum distance between two non-zero values in the convolution: M _i =max[M _i+ 1-2r _i ,M _i+ 1-2(M _i+1 -r _i ),r _i ],M _n =r _n ; (2) In the decoder, the features extracted by the encoder are up-sampled by sub-pixel convolution until the resolution is the same as the input image; the third layer network output of the encoder is connected to the first layer of the decoder using skips Perform feature stitching; use 1×1 convolution as the last layer of the decoder, so that the output is a single-channel image; (3) Normalize the gray value of the single-channel image to [-π,π]. 3. The method according to claim 1, wherein the training of the trained phase-dual resolution network comprises: (1) The constructed phase-dual resolution network f _net2 calculates and generates a pure phase hologram, and calculates the reconstructed image of the hologram according to the angular spectrum method

where the reconstruction distance is z; (2) Calculate the input target intensity map I and the reconstruction map calculated by the angular spectrum method respectively

between MS-SSIM loss and MSE loss; (3) The MS-SSIM loss and the MSE loss are summed according to a certain weight ratio as the overall consistency loss function to realize the simultaneous training of the phase-dual-resolution network models f _net1 and f _net2 . 4. The method according to claim 1, wherein, for the complex-valued wave field U _z , the calculation formula comprises:

5. The method according to claim 3, wherein the overall consistency loss function, the calculation formula comprises:

where I is the target amplitude,

represents the reconstructed amplitude,

used to calculate I and

The multi-scale structural similarity between , p represents the pixel, and is used to calculate the error pixel by pixel, and α = 0.84 is set empirically.

Images