CN115797231A

CN115797231A - Real-time hologram generation method based on neural network of Fourier inspiration

Info

Publication number: CN115797231A
Application number: CN202211548093.2A
Authority: CN
Inventors: 李燕; 凌玉烨; 董振兴; 徐超
Original assignee: Shanghai Jiaotong University
Current assignee: Shanghai Jiaotong University
Priority date: 2022-12-05
Filing date: 2022-12-05
Publication date: 2023-03-14

Abstract

A real-time hologram generating method based on a neural network inspired by Fourier transform adopts an improved U-Net network model with jump connection to fuse the spatial features from spatial branches and Fourier features from Fourier branches in an encoding stage, and converts an image into a hologram with a pure phase; the reconstruction result is obtained by angular spectroscopy in the decoding stage. The invention can be used for generating the real-time and high-fidelity phase hologram, and the reconstructed image has no speckle noise.

Description

Real-time hologram generation method based on neural network of Fourier heuristic

Technical Field

The invention relates to a technology in the field of image processing, in particular to a real-time high-quality hologram generation method based on an unsupervised Fourier inspired neural network.

Background

Holographic displays can provide pixel-level focus control and aberration correction, and are promising technologies for next-generation Virtual Reality (VR) and Augmented Reality (AR) optical devices. Computer Generated Holography (CGH) is a method of generating a holographic pattern by numerically simulating diffraction and interference of light. Conventional CGH algorithms can be classified into iterative methods and non-iterative methods. Up to now, conventional CGH algorithms always trade off computation time against display image quality.

Disclosure of Invention

Aiming at the defect that the high-fidelity hologram cannot be generated in real time due to the mutual restriction of the speed and the quality generated by the existing holography, the invention provides a real-time hologram generating method based on a neural network inspired by Fourier, which can be used for generating real-time and high-fidelity phase holograms and can ensure that the reconstructed image has no speckle noise.

The invention is realized by the following technical scheme:

the invention relates to a real-time hologram generating method of a neural network based on Fourier inspiration, which comprises the steps of adopting an improved U-Net network model with jump connection to fuse the spatial characteristics from a spatial branch and the Fourier characteristics from a Fourier branch in an encoding stage, and converting an image into a hologram with a pure phase; in the decoding stage, the reconstruction result is obtained by angular spectroscopy.

The improved U-Net network model comprises: two convolutional layers, four fourier blocks, and a hardtranh active layer, wherein: the two convolution layers extract spatial features, namely local feature maps, from the input images; each Fourier module converts the local feature map into a frequency domain by using two-dimensional fast Fourier transform (2D-FFT), extracts global Fourier features through two 1 x 1 convolution layers, converts the local feature map into an original domain by inverse Fourier transform (IFFT), and obtains a new feature map by adding the local feature map and the global feature map; the HardChanh active layer constrains the output hologram to a phase in the range [ - π, π ].

The improved U-Net network model adopts a Mean Square Error (MSE), a perception loss function and a Total Variation (TV) regularizer as loss functions to train so as to avoid checkerboard holograms, and specifically comprises the following steps:

wherein: a is a _gt Is the target image, phi is the net output phase-only hologram,

for the output of the l-layer of the pre-trained VGG-19, α is the corresponding weight and β is the penalty factor of the TV regularizer.

The angular spectrum method is as follows: complex amplitude image obtained by pure phase hologram propagation distance d

Wherein: phi x, y is the net output phase-only hologram, lambda is the wavelength, f _x And f _y Is the spatial frequency, d is the propagation distance between the SLM and the display plane,

in order to perform the fourier transformation, the method,

for spreading the factor, by

Obtaining a reconstructed image from the absolute values of

The invention relates to a system for realizing the method, which comprises the following steps: a training module and a reconstruction module, wherein: the training module learns global information in a frequency domain by utilizing Fourier transform on an input image to extract global features; and the reconstruction module performs angular spectrum propagation on the obtained hologram with the pure phase to obtain a complex amplitude image, and performs modulus operation on the complex amplitude image to obtain a reconstructed image, so that propagation from the hologram to the image is completed.

Technical effects

The invention integrally makes up the problems of poor reconstruction effect and generalization of the generated hologram in the existing neural network-based hologram generation method. Compared with the prior art, the method has the advantages that Fourier transformation is carried out on the input image, the space domain is converted into the frequency domain to obtain the global information, the overall characteristics are learned in the frequency domain and used for the generation process of the phase-only hologram, and the better visual effect and the better generalization result are realized through the reconstruction of the hologram.

Drawings

FIG. 1 is a flow chart of the present invention;

FIG. 2 is a schematic diagram of a real world optical reconstruction system;

FIG. 3 is a diagram illustrating the effect of an embodiment of simulation;

FIG. 4 is a diagram illustrating effects of an embodiment of the real world.

Detailed Description

As shown in fig. 1, the present embodiment relates to a real-time hologram generating method based on a neural network of fourier heuristic analysis, which specifically includes:

the first step, adopting a neural network model to fuse the space characteristics from the space branch and the Fourier characteristics from the Fourier branch, and converting an image into a hologram with a pure phase, specifically:

1.1 For each input image inputted, an image adjusted to 1080p by zero padding or cropping;

1.2 Respectively extracting features of the image through two times of 3 multiplied by 3 convolution, leakyReLU activation function and batch normalization operation;

1.3 For the features extracted in 1.2), sequentially passing through four down-sampling fourier modules, specifically including:

1.3.1 For 1.2), extracting local features in a spatial domain through two times of 3 multiplied by 3 convolution, leakyReLU activation function and batch normalization operation;

1.3.2 For the features extracted in 1.2), extracting global characteristics through two times of Fourier sub-block operation, and sequentially performing fast Fourier transform, 1 × 1 convolution, leakyReLU activation function, batch normalization operation and inverse fast Fourier transform;

1.3.3 Carrying out addition operation based on pixel points on the local features extracted in the step 1.3.1) and the global features extracted in the step 1.3.2) to obtain output with global features;

1.4 For the output with global characteristics obtained in the step 1.3), finally obtaining pure phase hologram output through 3 multiplied by 3 convolution and a HandChanh activation function in sequence;

and secondly, carrying out angular spectrum propagation on the pure phase hologram obtained in the first step to obtain a complex amplitude image, and then performing modulus operation on the complex amplitude image to obtain a finally reconstructed image.

In the first step, for fourier-inspired network training, 800 DIV2K datasets were selected as the training set. The expansion of the training data includes image horizontal flipping and rotation. The batch size used was 1, and the initial learning rate was 1 × 10 ^-3 . The network was trained using an AdamW optimizer with momentum of (0.9,0.999). The period of training is 40. A cosine decay strategy is then used to reduce the learning rate. Setting the layer number l as 5, and the weight alpha of the sensing loss as 2 \27960by 5 x 10 ^-2 The weight of TV regularizer β is 1 × 10 ^-3 . In the second step, the light source wavelength is 543nm and the propagation distance is 7cm for angular spectral propagation of the hologram. The SLM resolution is set to 1080 × 1920 with a corresponding pixel pitch of 3.74 μm. All experiments were trained and tested using NVIDIA GeForceRTX3090GPU card.

Example 1

The present example uses peak signal-to-noise ratio (PSNR) and Structural Similarity Index (SSIM) to evaluate the performance of the present invention and other methods. As shown in Table 1, in the numerical simulation reconstruction, the PSNR and SSIM of the invention are respectively 29.16dB and 0.935, which are respectively improved by 4.97dB and 0.17 compared with the U-Net method. Although slightly worse than SGD performance using the 500 iteration method, the present invention is 1585 times higher in hologram generation time than the SGD method. The visual effect is as shown in fig. 2, and the reconstructed image obtained by the method has higher resolution, can see more details and is superior to other reconstruction methods.

Example 2

Compared with embodiment 1, the optical reconstruction is performed in a real-world optical system to obtain a pure phase hologram, as shown in fig. 2, which is a schematic diagram of the optical reconstruction apparatus of this embodiment, in which the SLM is HOLOEYEGAEA-2-VIS-036, the resolution is 3840 × 2160, and the pixel pitch is 3.74 μm. To match the resolution of the SLM, the hologram is adjusted to 4K by nearest neighbor interpolation. The settings of other parameters in the experiment are consistent with the simulation result. An optional 4F system is provided to filter out higher order diffraction artifacts. The final reconstructed image was taken by a canon EOSM10 camera.

The present embodiment uses peak signal-to-noise ratio (PSNR) and Structural Similarity Index (SSIM) to evaluate the performance of the present invention and other methods. The visual effect is shown in fig. 4. Although the GS method provides high contrast, it suffers from the most severe speckle noise. The DPAC method exhibits low contrast because its checkerboard pattern creates ghost copies and results in a loss of intensity in the central region. SGD has a good balance between image contrast and speckle noise, but still suffers from speckle noise in some scenes, and cannot generate high-quality holograms in real time. The deep learning method U-Net can only provide low contrast, low resolution reconstructed images. The optical reconstruction image obtained by the method has better contrast and less speckle noise, and is superior to other reconstruction methods.

The examples were compared with the process of the present invention using the following prior art:

GS:《Gerchberg RW.A practical algorithm for the determination of plane from image and diffraction pictures[J].Optik,1972,35(2):237-246.》

DPAC:《Maimone A,Georgiou A,Kollin J S.Holographic near-eye displays for virtual and augmented reality[J].ACM Transactions on Graphics(Tog),2017,36(4):1-16.》

SGD:《Peng Y,Choi S,Padmanaban N,et al.Neural holography with camera-in-the-loop training[J].ACM Transactions on Graphics(TOG),2020,39(6):1-14.》

UNet《WuJ,Liu K,Sui X,et al.High-speed computer-generated holography usingan autoencoder-based deep neural network[J].Optics Letters,2021,46(12):2908-2911.》

TABLE 1

Algorithm	PSNR(dB)	SSIM	Time(s)
				GS	23.41	0.661	3.412
DPAC	26.73	0.852	0.001
				SGD	32.03	0.941	23.78
U-Net	24.19	0.765	0.007
				The invention	29.16	0.935	0.015

Compared with the prior art, the simulation results of reconstruction after obtaining the hologram by the method are shown in table 1 and fig. 3, and it can be seen that compared with the convolution-based U-Net method, the method can obtain higher image resolution and image fidelity, and compared with the iterative method, the generation speed of the 1080p hologram is only 0.015s, which is 1585 times of that of the iterative method SGD, so that the real-time generation of the high-fidelity pure phase hologram is really realized. The result of the real world experiment is shown in fig. 4, and similar to the simulation result, the hologram generated by the invention can reconstruct a clearer image effect.

The foregoing embodiments may be modified in many different ways by those skilled in the art without departing from the spirit and scope of the invention, which is defined by the appended claims and all changes that come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.

Claims

1. A real-time hologram generating method based on a neural network inspired by Fourier is characterized in that an improved U-Net network model with jump connection is adopted in a coding stage to fuse spatial features from spatial branches and Fourier features from Fourier branches, and an image is converted into a phase-only hologram; in the decoding stage, the reconstruction result is obtained by angular spectroscopy.

2. The method of claim 1, wherein the improved U-Net network model comprises: two convolutional layers, four fourier blocks, and a hardtan activation layer, wherein: the two convolution layers extract spatial features, namely local feature maps, from the input images; each Fourier module converts the local feature map into a frequency domain by using two-dimensional fast Fourier transform (2D-FFT), extracts global Fourier features through two 1 x 1 convolution layers, converts the local feature map into an original domain by inverse Fourier transform (IFFT), and obtains a new feature map by adding the local feature map and the global feature map; the HardChanh active layer constrains the output hologram to a phase in the range [ - π, π ].

3. The method for generating a real-time hologram based on a neural network of fourier heuristic claims 1 or 2, characterized in that the improved U-Net network model is trained to avoid checkerboard holograms using Mean Square Error (MSE), perceptual loss function and Total Variation (TV) regularizer as loss function, in particular:

wherein: a is _gt Is the target image, phi is the net output phase-only hologram,

4. The method of claim 1, wherein the angular spectrum method comprises: complex amplitude image obtained by phase-only hologram propagation distance d

Wherein: phi (x, y) is the phase-only hologram output by the network, lambda is the wavelength, f _x And f _y Is the spatial frequency, d is the propagation distance between the SLM and the display plane,

in order to carry out the Fourier transform,

for spreading the factor, by

Obtaining a reconstructed image from the absolute values of

5. A method for generating a real-time hologram based on a neural network of Fourier heuristics according to any one of claims 1-4, comprising in particular:

1.1 For each input image that is input, an image adjusted to 1080p by zero padding or cropping;

1.2 Respectively extracting the characteristics of the image through two times of 3 multiplied by 3 convolution, leakyReLU activation function and batch normalization operation;

1.3.3 Performing pixel-based addition operation on the local features extracted in the step 1.3.1) and the global features extracted in the step 1.3.2) to obtain output with global features;

1.4 For the output with global characteristics obtained in the step 1.3), finally obtaining pure phase hologram output through 3 multiplied by 3 convolution and a HandTanh activation function in sequence;

6. A real-time hologram generation system of a neural network based on Fourier heuristics implementing the method of any one of claims 1-4, comprising: a training module and a reconstruction module, wherein: the training module learns global information in a frequency domain by utilizing Fourier transform on an input image to extract global features; and the reconstruction module performs angular spectrum propagation on the obtained hologram with the pure phase to obtain a complex amplitude image, and performs modulus operation on the complex amplitude image to obtain a reconstructed image, so that propagation from the hologram to the image is completed.