CN114353946B - Diffraction snapshot spectrum imaging method - Google Patents

Abstract

A diffraction snapshot spectral imaging method disclosed by the invention belongs to the field of computational photography. The present invention is applied to the diffraction snapshot spectral imaging system, and jointly optimizes the encoding structure of the diffractive optical element and the reconstruction and decoding neural network in a data-driven manner, and obtains a relatively optimal optical encoding structure and its corresponding calculation and decoding model, so that the encoding and The decoding part is more in line with each other; the optical coding structure considers the quantitative requirements in actual manufacturing during the optimization stage, so that the structure of the optimized diffractive optical element is consistent with the structure of the manufactured diffractive optical element, improving the accuracy of the reconstructed image; using a diffractive optical element with a high degree of coding freedom As the coding part, the optical element has the advantages of small size and compact structure, and has the ability of real-time imaging. A differentiable model is used to model the entire encoding and decoding process, and then the model can be realized and optimized in any machine learning automatic differentiation framework, thereby improving the versatility of the present invention.

Description

A Diffraction Snapshot Spectral Imaging Method

Technical Field

The invention relates to a diffraction snapshot spectral imaging method, in particular to a method capable of acquiring high-quality hyperspectral images in a snapshot manner, and belongs to the field of computational photography.

Background Art

Hyperspectral imaging technology is a technology that combines spatial imaging technology with spectral imaging technology. It can simultaneously record the spatial and spectral information of an object. Compared with traditional imaging technology, it can obtain richer object details. It has been widely used in many fields such as agricultural production, material identification, geological exploration and biomedicine. The traditional snapshot spectral imaging method usually uses multiple refractive lenses or reflective lenses to achieve optical encoding, and reconstructs the required spectral image by computational decoding. The physical system corresponding to this method has a complex optical encoding part, which makes its volume usually large. Thanks to the high degree of freedom encoding capability of diffractive optical elements, the diffractive snapshot spectral imaging method uses a diffractive optical element to replace the refractive lens or reflective lens used in the previous system as the optical encoding structure, which greatly reduces the volume of the entire system. However, the existing diffractive spectral imaging method usually manually designs the structure of the diffractive optical element heuristically. This design method requires the use of effective prior knowledge for manual design or optimization solution, and it is difficult to ensure that the final designed structure is a global optimal solution. In addition, the diffractive optical element structure obtained by the existing design method is a high-precision, nearly continuous structure, while the physical manufacturing process can usually only process diffractive optical elements with no more than 16 steps, which makes the optimized structure different from the actual manufactured structure, and further leads to the reconstruction algorithm designed during the optimization being unable to reconstruct the spectral image with high quality.

Summary of the invention

To solve the problems of complex optical coding optimization in the existing diffraction snapshot spectral imaging method, and the difference between the optimized diffraction optical element structure and the actual physical manufacturing structure. The technical problem to be solved by a diffraction snapshot spectral imaging method disclosed in the present invention is: to jointly optimize the coding structure and the reconstruction decoding neural network of the diffraction optical element in the diffraction snapshot spectral imaging system in a data-driven manner to obtain a relatively optimal optical coding structure. At the same time, the optical coding structure considers the quantitative requirements in actual manufacturing during the optimization stage, so that the optimized diffraction optical element structure is consistent with the manufactured diffraction optical element structure, thereby improving the reconstructed image accuracy of the entire spectral imaging system. In addition, the present invention adopts a diffraction optical element with a high degree of coding freedom as the coding part, and the imaging system using the present invention has the advantages of small size and compact structure compared to the imaging system using a variety of refractive or reflective lenses as the coding part.

To achieve the above object, the present invention adopts the following technical solutions.

The present invention discloses a diffraction snapshot spectral imaging method, which is applied to a diffraction snapshot spectral imaging system, jointly optimizes the optical coding structure and reconstruction algorithm of the diffraction snapshot spectral imaging system, and simultaneously searches for the optical coding structure most suitable for the spectral imaging task and its corresponding computational decoding model in a data-driven manner; the present invention models the optical physical coding part of the system using a point spread function model, and splices the reconstruction decoder with it to form a complete end-to-end model for joint optimization and solution; at the same time, in order to ensure that the diffraction optical element structure obtains a quantized step-shaped structure after optimization, the present invention uses an adaptive quantization perception training method to process the height map of the diffraction optical element, so that the optimized diffraction optical element structure is consistent with the manufactured diffraction optical element structure, thereby improving the reconstructed image accuracy of the entire spectral imaging system; after the training is completed, the optimized diffraction optical element design is directly used for manufacturing, and a diffraction snapshot spectral imaging physical system is built; in the built diffraction snapshot spectral imaging physical system, the diffraction optical element phase modulates the incident light, the RGB image sensor collects the modulated light field, and the trained convolutional neural network reconstructs and decodes the collected RGB image to obtain the final required hyperspectral image. The present invention, while maintaining the advantage of the diffraction snapshot spectral imaging system being small and compact, jointly optimizes the coding structure and reconstruction decoding neural network of the diffraction optical element in the diffraction snapshot spectral imaging system in a data-driven manner, and obtains a relatively optimal optical coding structure and its corresponding computational decoding model, thereby making the coding and decoding parts more compatible with each other; at the same time, the present invention considers the quantitative requirements in actual manufacturing in the optimization stage of the optical coding structure, so that the optimized diffraction optical element structure is consistent with the manufactured diffraction optical element structure, thereby improving the reconstructed image accuracy of the entire spectral imaging system, making the optimized diffraction optical element structure consistent with the manufactured diffraction optical element structure, and further improving the spectral image reconstruction accuracy. The present invention is used in multiple fields involving spectral imaging, such as manned spaceflight, geological survey, agricultural production, and biomedicine.

The present invention discloses a diffraction snapshot spectral imaging method, comprising the following steps:

Step 101: Construct an optical physics encoding forward model according to diffraction optics theory.

The optical physical encoding forward model in step 101 is constructed based on the point spread function model. According to the direction of the incident light, when a wave field of a point light source of a natural scene with a wavelength of λ propagates a certain distance d and reaches the plane position (x, y) in front of the diffractive optical element of the diffractive optical encoder, it is expressed as U ₀ :

Where i is a complex unit. The diffractive optical element will modulate the phase of the wave field, and the modulated wave field U1 is expressed as:

Where A(x, y) is the aperture function. If its value is 0, it means that the propagation at this position is blocked, and if its value is 1, it means that the propagation at this position is allowed. φ(x, y, λ) is the phase shift generated by the diffractive optical element at the plane position (x, y) for the wave field with a wavelength of λ:

φ(x, y, λ)=(n _λ -1)H(x, y) (3)

Where _nλ is the refractive index of the diffractive optical element for wavelength λ light, and H(x, y) is the height of the diffractive optical element design structure at the plane position (x, y). The wave field continues to propagate a distance z and reaches the RGB image sensor plane, which is represented by U ₂ :

Where _fx and _fy are the frequency variables corresponding to the spatial position (x, y), respectively. The point spread function P is derived from the wave field on the sensor plane, which is the square of the complex wave field on the sensor plane:

P(x,y,λ)∝|U ₂ (x,y,λ)| ² (5)

After obtaining the point spread function, the target scene image I(x, y, λ, d) corresponding to the wavelength λ is convolved with the corresponding point spread function P(x, y, λ) respectively:

That is, the modulated image I′(x, y, λ) corresponding to the wavelength and scene depth is obtained; the obtained modulated scene I′ contains the modulated image I′ _λ,d in each case of the full set of wavelengths Λ and the full set of depths D:

I＝{I′ _{λ, d} |λ∈Λ, d∈D} (7)

After the modulated image is collected by the RGB image sensor according to its spectral response curve R _c in the range of [λ ₀ ,λ ₁ ], the observed modulated RGB image I _c∈{R,G,B} (x,y) is obtained:

Where η is the sensor noise. So far, the forward model of optical coding is established as shown in formula (1-8).

Step 102: Perform quantization perception processing on the diffraction optical element height map in the optical physical coding forward model constructed in step 1, so that it presents a quantized step pattern.

In step 102, the height map of the diffractive optical element is subjected to quantization perception processing, and the height map after quantization perception is expressed as:

H _aq =α×F(Q(H _f ))+(1-α)×H _f (9)

Among them, _Hf is the original height map weight of full precision, α is the quantization perception control parameter, which changes with the number of training steps, and _T0 and _T1 correspond to the training steps at the beginning and end of quantization perception respectively:

Q is the quantization operation function, which quantizes the input height map H into L steps within h _max :

F is the adaptive fine-tuning function:

F(Q( _Hf ))＝Q( _Hf )+ _Wl (12)

It dynamically adjusts W _l according to the adaptive quantization loss to adjust the physical height of each uniform step to an optimal quantization form.

Step 103: construct a computational decoding model based on a differentiable neural network, and use the computational decoding model to predict and infer the original spectral image to reconstruct the image.

即计算解码模型被表示为：The computational decoding model in step 103 is an arbitrary differentiable neural network structure module N, which predicts and infers the original spectral image reconstructed from the image I _{c∈{R, G, B}} (x, y) modulated by the optical encoding module

That is, the computational decoding model is expressed as:

Step 104: The coding structure and the reconstruction decoding neural network of the diffractive optical element in the diffraction snapshot spectral imaging system are jointly optimized in a data-driven manner to obtain a relatively optimal optical coding structure and its corresponding computational decoding model; at the same time, in the optimization stage, the coding structure of the diffractive optical element is made to present a quantized step pattern through the quantization perception processing in step 102, so as to consider the quantization requirements in actual manufacturing in the optimization stage, so that the optimized diffractive optical element structure is consistent with the manufactured diffractive optical element structure, thereby improving the reconstructed image accuracy of the entire spectral imaging system.

The implementation method of step 104 is: respectively set the learning rate, weight initialization method, and weight decay coefficient for the optical physical coding forward model constructed in step 101 and the computational decoding model constructed in step 103, and set the batch size, optimization method, number of iterations of the joint optimization, and the parameters for starting and ending the quantization perception processing in step 102. The output of the optical physical coding forward model constructed in step 101 is used as the input of the computational decoding model constructed in step 103, and the constructed joint coding and decoding spectral imaging model is end-to-end jointly optimized and trained to obtain a relatively optimal optical coding structure and its corresponding computational decoding model; at the same time, the optical coding structure takes into account the quantization requirements in actual manufacturing during the optimization stage, so that the optimized diffractive optical element structure is consistent with the manufactured diffractive optical element structure, thereby improving the reconstructed image accuracy of the entire spectral imaging system.

为包含重建图像损失部分、自适应量化部分和解码权重的L2正则化损失部分：The end-to-end joint optimization in step 104 can be implemented using any machine learning automatic differentiation framework. The end-to-end loss function used in the end-to-end joint optimization is

It is the L2 regularization loss part that includes the reconstructed image loss part, the adaptive quantization part and the decoding weight:

是光谱重建损失函数，

是自适应量化损失，ω是重建模块的网络权重，β和γ为比例参数。in,

is the spectral reconstruction loss function,

is the adaptive quantization loss, ω is the network weight of the reconstruction module, and β and γ are scaling parameters.

由重建图像

和目标光谱图像I之间的平均绝对误差MAE计算得到：Spectral reconstruction loss function

Reconstructed image

The mean absolute error MAE between the target spectral image I is calculated as:

Where K is the number of pixels in the image.

表示为量化操作后的高度图和原始高度图之间的均方误差MSE：Adaptive Quantization Loss

It is expressed as the mean square error MSE between the height map after quantization and the original height map:

Where J is the number of pixels in the height map;

It also includes step 105 and step 106:

Step 105: A diffractive optical element is manufactured according to the optical coding structure optimized in step 104, and a diffractive snapshot spectral imaging system is constructed using the diffractive optical element and an RGB image sensor. Since the diffractive optical element has a high degree of coding freedom, the volume of the constructed diffraction snapshot spectral imaging system can be reduced, thereby improving the compactness of the diffraction snapshot spectral imaging system.

Step 106: Use the diffraction snapshot spectral imaging system built in step 105 to photograph the target scene, and use the computational decoding model optimized in step 104 to decode and reconstruct the observation image photographed by the physical system to obtain the required hyperspectral image corresponding to the target scene, thereby improving the reconstructed image accuracy of the entire spectral imaging system.

Preferably, a GPU is used to perform end-to-end joint optimization training on the joint encoding and decoding spectral imaging model constructed in step 104 to obtain a relatively optimal optical coding structure and its corresponding computational decoding model.

Beneficial effects:

1. A diffraction snapshot spectral imaging method disclosed in the present invention jointly optimizes the coding structure and reconstruction decoding neural network of the diffraction optical element in the diffraction snapshot spectral imaging system in a data-driven manner to obtain a relatively optimal optical coding structure and its corresponding computational decoding model, thereby making the coding and decoding parts more compatible with each other.

2. In a diffraction snapshot spectral imaging method disclosed in the present invention, the optical encoding structure takes into account the quantitative requirements in actual manufacturing during the optimization stage, so that the optimized diffraction optical element structure is consistent with the manufactured diffraction optical element structure, thereby improving the reconstructed image accuracy of the entire spectral imaging system.

3. A diffraction snapshot spectral imaging method disclosed in the present invention adopts a diffraction optical element with high coding freedom as the encoding part. Compared with an imaging system using a variety of refractive or reflective lenses as the encoding part, it has the advantages of small size and compact structure. It can perform high-spectral imaging without scanning in spatial or spectral dimensions and has the ability of real-time imaging.

4. A diffraction snapshot spectral imaging method disclosed in the present invention adopts a differentiable model to model the entire encoding and decoding process, and then the model can be implemented and optimized in any machine learning automatic differentiation framework, thereby improving the versatility of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments.

FIG1 is a flow chart of a diffraction snapshot spectral imaging method provided by an embodiment of the present invention;

FIG2 is a schematic diagram of the physical system structure of the diffraction snapshot spectral imaging method provided by an embodiment of the present invention;

FIG3 is a joint training framework diagram of the diffraction snapshot spectral imaging method provided by an embodiment of the present invention;

FIG4 is a schematic diagram of quantitative perception processing of a diffraction snapshot spectral imaging method provided by an embodiment of the present invention;

FIG5 is a diagram of a reconstruction network structure of a diffraction snapshot spectral imaging method provided by an embodiment of the present invention;

In the figure: 101 - diffractive optical element, 102 - aperture, 103 - clamping member, 104 - RGB image sensor.

DETAILED DESCRIPTION

In order to better illustrate the purpose and advantages of the present invention, the invention is further described below with reference to the accompanying drawings and examples.

Embodiment 1:

As shown in FIG2 , an imaging system corresponding to a diffraction snapshot spectroscopy method disclosed in this embodiment includes a diffractive optical element 101, an aperture 102, a clamp 103, and an RGB image sensor 104. The aperture 102 is used to block unnecessary light. The diffractive optical element 101 is used to modulate the phase of incident light. The clamp 103 is used to fix the diffractive optical element on the sensor. The RGB image sensor 104 is used to detect the modulated image.

FIG2 shows a schematic diagram of the physical structure of an imaging system corresponding to the diffraction snapshot spectroscopy method provided by an embodiment of the present invention. According to the light path exit direction, the incident light passes through the aperture 102 and the diffractive optical element 101 in sequence, and is finally collected by the RGB image sensor 104 and converted into an image digital signal. The image digital signal is output to the spectral reconstruction network and image processing is performed through the spectral reconstruction network to realize spectral imaging. Spectral imaging is realized by performing image processing through the spectral reconstruction network. The implementation method is as follows: the RAW image collected by the RGB image sensor is subjected to a linear de-mosaicing algorithm to obtain an RGB image, and then the de-mosaiced RGB image is cropped into an RGB image with a width and height of 2 to the power of n, and then the image is input into a pre-trained reconstruction network. The image output by the reconstruction network is the desired spectral image.

As shown in FIG1 , a diffraction snapshot spectral imaging method disclosed in this embodiment includes the following steps:

Step 101: Construct an optical physics encoding forward model according to diffraction optics theory.

The optical physical encoding forward model in step 101 is constructed based on the point spread function model. First, according to the direction of the incident light, when a wave field of a point light source of a natural scene with a wavelength of λ propagates a certain distance d and reaches the plane position (x, y) in front of the diffractive optical element of the diffractive optical encoder of the system of the present invention, it is expressed as U ₀ :

Where i is a complex unit, d is set to 1m, and the wavelength λ sampling points include wavelengths with an interval of 10nm from 400nm to 700nm; the diffractive optical element will modulate the phase of the wave field, and the modulated wave field U ₁ is expressed as:

Where A(x, y) is the aperture function. If its value is 0, it means that the propagation at this position is blocked, and if its value is 1, it means that the propagation at this position is allowed. φ(x, y, λ) is the phase shift generated by the diffractive optical element at the plane position (x, y) for the wave field with a wavelength of λ:

φ(x, y, λ)=(n _λ -1)H(x, y) (3)

Where _nλ is the refractive index of the diffractive optical element for the wavelength λ light, and the material of the embodiment is OHARA SK1300 quartz glass material; H(x, y) is the height of the diffractive optical element design structure at the plane position (x, y). Then the wave field continues to propagate a distance z and reaches the RGB image sensor plane, which can be expressed as U ₂ :

Where _fx and _fy are the frequency variables corresponding to the spatial position (x, y), respectively, and the distance z from the diffractive optical element to the RGB image sensor is set to 50 mm; then the point spread function P is obtained, which is the square size of the complex wave field on the sensor plane:

P(x,y,λ)∝|U ₂ (x,y,λ)| ² . (5)

After obtaining the point spread function, the target scene image I(x, y, λ, d) corresponding to the wavelength λ is convolved with the corresponding point spread function P(x, y, λ) respectively:

The modulated image I′(x, y, λ) corresponding to the wavelength and scene depth can be obtained; the modulated scene I′ finally obtained contains the modulated image I′ _λ,d in each case of the full set of wavelengths Λ and full set of depths D:

I＝{I′ _{λ, d} |λ∈Λ, d∈D} (7)

After the modulated image is collected by the RGB image sensor according to its spectral response curve R _c in the range of [λ ₀ ,λ ₁ ], the observed modulated RGB image I _c∈{R,G,B} (x,y) is obtained:

Where η is the sensor noise, which is set to Gaussian noise with a mean of 0 and a standard deviation of 0.001; the RGB sensor model used in the embodiment is FLIR GS3-U3-41S4C; So far, the forward model of optical coding is established according to the above relationship. And the above model is implemented in TensorFlow2.

Step 102: Perform quantization perception processing on the diffraction optical element height map in the optical physical coding forward model constructed in step 1, so that it presents a quantized step pattern.

In step 102, the height map of the diffractive optical element is subjected to quantization perception processing, and the height map after quantization perception can be expressed as:

H _aq =α×F(Q(H _f ))+(1-α)×H _f (9)

Where _Hf is the full-precision original height map weight, α is the quantized perception control parameter, which changes with the number of training steps, and _T0 and _T1 are set to the number of training steps corresponding to the 5th training round and the 40th training round, respectively:

Q is the quantization operation function, which quantizes the input height map H into a style of L steps within h _max , and the number of steps L is 4:

F is the adaptive fine-tuning function:

F(Q( _Hf ))＝Q( _Hf )+ _Wl (12)

It dynamically adjusts W _l according to the adaptive quantization loss to adjust the physical height of each uniform step to an optimal quantization form.

Step 103: construct a computational decoding model based on a differentiable neural network, and use the computational decoding model to predict and infer the original spectral image to reconstruct the image.

即可被表示为：The computational decoding model in step 103 is Res-UNet, denoted as N, which predicts and infers the original spectral image reconstructed from the image I _{c∈{R, G, B}} (x, y) modulated by the optical encoding module

It can be expressed as:

Step 104: The coding structure and the reconstruction decoding neural network of the diffractive optical element in the diffraction snapshot spectral imaging system are jointly optimized in a data-driven manner to obtain a relatively optimal optical coding structure and its corresponding computational decoding model; at the same time, in the optimization stage, the coding structure of the diffractive optical element is made to present a quantized step pattern through the quantization perception processing in step 102, so as to consider the quantization requirements in actual manufacturing in the optimization stage, so that the optimized diffractive optical element structure is consistent with the manufactured diffractive optical element structure, thereby improving the reconstructed image accuracy of the entire spectral imaging system.

In step 104, the learning rates of the optical physical coding forward model and the computational decoding model are initialized to 10-2 and 10-3 respectively, and the learning rates of both are reduced to 0.8 times of the original value for each training round on the training data set. In step 104, the batch size is set to 4, indicating the number of images processed by a single iterative optimization; the optimization method is Adam, and the number of iterations is 50 rounds; and the quantization-aware training is set to start at the 5th round and end at the 40th round.

为包含重建图像损失部分、自适应量化部分和解码权重的L2正则化损失部分：The end-to-end joint optimization in step 104 is implemented using the TensorFlow2 automatic differentiation framework, and the dataset uses the ICVL spectral scene dataset. The end-to-end loss function used in the end-to-end joint optimization is

It is the L2 regularization loss part that includes the reconstructed image loss part, the adaptive quantization part and the decoding weight:

是光谱重建损失函数，

是自适应量化损失，ω是重建模块的网络权重，比例参数β和γ分别取0.01和0.0001。in,

is the spectral reconstruction loss function,

is the adaptive quantization loss, ω is the network weight of the reconstruction module, and the scaling parameters β and γ are taken as 0.01 and 0.0001 respectively.

由重建图像

和目标光谱图像I之间的平均绝对误差MAE计算得到：Spectral reconstruction loss function

Reconstructed image

The mean absolute error MAE between the target spectral image I is calculated as:

Where K is the number of pixels in the image.

表示为量化操作后的高度图和原始高度图之间的均方误差MSE：Adaptive Quantization Loss

It is expressed as the mean square error MSE between the height map after quantization and the original height map:

Where J is the number of pixels in the height map;

Step 105: A diffractive optical element is manufactured according to the optical coding structure optimized in step 104, and a diffractive snapshot spectral imaging system is constructed using the diffractive optical element and an RGB image sensor. Since the diffractive optical element has a high degree of coding freedom, the volume of the constructed diffraction snapshot spectral imaging system can be reduced, thereby improving the compactness of the diffraction snapshot spectral imaging system.

The physical parameters of the real object in step 105 are consistent with all the physical parameters in the simulation optimization stage, that is, the distance from the shooting scene to the diffractive optical element is 1 m, and the distance from the diffractive optical element to the sensor is 50 mm.

Step 106: Use the diffraction snapshot spectral imaging system built in step 105 to shoot the target scene, and use the computational decoding model Res-UNet optimized in step 104 to decode and reconstruct the observation image shot by the physical system to obtain the required hyperspectral image corresponding to the target scene, thereby improving the reconstructed image accuracy of the entire spectral imaging system. At this point, diffraction snapshot spectral imaging is achieved.

To illustrate the effect of the present invention, this embodiment will compare the joint optimization encoding and decoding method including Fresnel lens encoding CASSI encoding, conventional joint optimization encoding and decoding (Deep Optics) and the diffraction snapshot spectral imaging of this example under the same experimental conditions.

1. Experimental conditions

The hardware test conditions of this experiment are as follows: the CPU uses Intel(R) Xeon(R) Gold 5218R, the running memory size is 256GB, the GPU is NVIDIA GeForce RTX3090, the GPU memory is 24G, and the CUDA computing library version is 11.0; the hyperspectral images used in the test are from the ICVL dataset, and the input image size is 512×512; the test methods are all implemented using TensorFlow 2.5. In order to ensure the fairness of the comparative experiment, the physical parameters and reconstruction network structures of all models using different encoding methods are consistent with the physical parameters and reconstruction network structures introduced in this example.

2. Experimental results

In order to quantitatively measure the quality of the reconstruction results, Peak Signal to Noise Ratio (PSNR) and Structural Similarity (SSIM) are used to measure the spatial quality and visual effect of the reconstruction results; Root Mean Square Error (RMSE) and Relative Dimensionless Global Error in Synthesis (ERGAS) are used to measure the spectral fidelity of the reconstruction results;

Table 1 Comparative experimental spectral reconstruction index results of different snapshot spectral imaging methods

Methods\Indicators PSNR SSIM RMSE ERGAS Fresnel 27.41 0.869 0.0556 32.18 CASSI 30.66 0.899 0.0354 20.74 Deep Optics 31.68 0.939 0.0319 19.16 Method disclosed in the embodiment of the present invention 35.32 0.968 0.0205 12.35

Table 1 shows the spectral reconstruction index results of different snapshot spectral imaging methods on the test set after training on the ICVL dataset. It can be seen that the method disclosed in the embodiment of the present invention has better performance in terms of both spatial image quality and spectral fidelity.

In summary, the diffraction snapshot spectral imaging method disclosed in this embodiment uses a diffraction optical element as the optical encoding part, which significantly reduces the volume and complexity of the physical part; at the same time, the method proposed in this embodiment uses the constructed simulation model to train in the automatic differentiation framework to jointly optimize the design of the codec based on a data-driven model, and does not require additional prior knowledge or manual participation in the design to make the codec algorithms fit each other, and this method can usually find a globally optimal codec design solution in its solution space; in addition, the diffraction snapshot spectral imaging method proposed in this embodiment also uses a quantitative perception method in the codec joint optimization stage to consider the quantitative requirements of actual physical manufacturing, so that the difference between simulation and real objects is reduced, thereby achieving a better spectral reconstruction effect. The spectral imaging method proposed in this embodiment has important application value in low-light fields such as remote sensing and biomedicine, and high-speed measurement fields such as dynamic object monitoring and tracking.

The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention patent may include various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims (8)

1. A diffraction snapshot spectral imaging method, characterized in that it comprises the following steps: Step 101: construct an optical physics encoding forward model according to diffraction optics theory; Step 102: performing quantization perception processing on the diffractive optical element height map in the optical physical coding forward model constructed in step 1, so that it presents a quantized step pattern; Step 103: constructing a computational decoding model based on a differentiable neural network, and using the computational decoding model to predict and infer the original spectral image to reconstruct the image; Step 104: The coding structure and the reconstruction decoding neural network of the diffractive optical element in the diffraction snapshot spectral imaging system are jointly optimized in a data-driven manner to obtain a relatively optimal optical coding structure and its corresponding computational decoding model; at the same time, in the optimization stage, the coding structure of the diffractive optical element is made to present a quantized step pattern through the quantization perception processing in step 102, so as to consider the quantization requirements in actual manufacturing in the optimization stage, so that the optimized diffractive optical element structure is consistent with the manufactured diffractive optical element structure, thereby improving the reconstructed image accuracy of the entire spectral imaging system. 2. The diffraction snapshot spectral imaging method according to claim 1, characterized in that it further comprises step 105 and step 106: Step 105: manufacturing a diffractive optical element according to the optical coding structure optimized in step 104, and constructing a diffractive snapshot spectral imaging system using the diffractive optical element and an RGB image sensor. Since the diffractive optical element has a high degree of coding freedom, the volume of the constructed diffractive snapshot spectral imaging system can be reduced, thereby improving the compactness of the diffractive snapshot spectral imaging system. Step 106: Use the diffraction snapshot spectral imaging system built in step 105 to photograph the target scene, and use the computational decoding model optimized in step 104 to decode and reconstruct the observation image photographed by the physical system to obtain the required hyperspectral image corresponding to the target scene, thereby improving the reconstructed image accuracy of the entire spectral imaging system. 3. A diffraction snapshot spectral imaging method according to claim 1 or 2, characterized in that: the optical physical encoding forward model in step 101 is constructed based on a point spread function model; according to the direction of the incident light, when a wave field of a point light source of a natural scene with a wavelength of λ propagates a certain distance d and reaches the plane position (x, y) in front of the diffractive optical element of the diffractive optical encoder, it is expressed as U ₀ :

Where i is a complex unit; the diffractive optical element will modulate the phase of the wave field, and the modulated wave field U ₁ is expressed as:

Where A(x, y) is the aperture function. If its value is 0, it means that the propagation at this position is blocked, and if its value is 1, it means that the propagation at this position is allowed. φ(x, y, λ) is the phase shift generated by the diffractive optical element at the plane position (x, y) for the wave field with a wavelength of λ: φ(x, y, λ)=(n _λ -1)H(x, y) (3) Where _nλ is the refractive index of the diffractive optical element for the wavelength λ light, H(x, y) is the height of the diffractive optical element design structure at the plane position (x, y); the wave field continues to propagate a distance z and reaches the RGB image sensor plane, which is represented by U ₂ :

where _fx and _fy are the frequency variables corresponding to the spatial position (x, y), respectively; the point spread function P is derived from the wave field on the sensor plane, which is the square of the complex wave field on the sensor plane: P(x,y,λ)∝|U ₂ (x,y,λ)| ² (5) After obtaining the point spread function, the target scene image I(x, y, λ, d) corresponding to the wavelength λ is convolved with the corresponding point spread function P(x, y, λ) respectively:

That is, the modulated image I′(x, y, λ) corresponding to the wavelength and scene depth is obtained; the obtained modulated scene I′ contains the modulated image I′ _λ,d in each case of the full set of wavelengths Λ and the full set of depths D: I＝{I′ _{λ, d} |λ∈Λ, d∈D} (7) After the modulated image is collected by the RGB image sensor according to its spectral response curve R _c in the range of [λ ₀ ,λ ₁ ], the observed modulated RGB image I _c∈{R,G,B} (x,y) is obtained:

Where η is the sensor noise; at this point, the forward model of optical coding is established as shown in formula (1-8). 4. A diffraction snapshot spectral imaging method according to claim 3, characterized in that: in step 102, the height map of the diffractive optical element is subjected to quantized perception processing, and the height map after quantized perception is expressed as: H _aq =α×F(Q(H _f ))+(1-α)×H _f (9) Among them, _Hf is the original height map weight of full precision, α is the quantization perception control parameter, which changes with the number of training steps, and _T0 and _T1 correspond to the training steps at the beginning and end of quantization perception respectively:

Q is the quantization operation function, which quantizes the input height map H into L steps within h _max :

F is the adaptive fine-tuning function: F(Q( _Hf ))＝Q( _Hf )+ _Wl (12) It dynamically adjusts W _l according to the adaptive quantization loss to adjust the physical height of each uniform step to an optimal quantization form. 5. A diffraction snapshot spectral imaging method as claimed in claim 4, characterized in that: the computational decoding model in step 103 is an arbitrarily differentiable neural network structure module N, which predicts and infers the original spectral image reconstructed image from the image I _{c∈{R, G, B}} (x, y) modulated by the optical encoding module

That is, the computational decoding model is expressed as:

6. A diffraction snapshot spectral imaging method as described in claim 5, characterized in that: the implementation method of step 104 is: setting the learning rate, weight initialization method, and weight decay coefficient for the optical physical coding forward model constructed in step 101 and the computational decoding model constructed in step 103, respectively, and setting the batch size, optimization method, number of iterations of the joint optimization, and the parameters for starting and ending the quantization perception processing of step 102; using the output of the optical physical coding forward model constructed in step 101 as the input of the computational decoding model constructed in step 103, and constructing a joint coding and decoding spectral imaging model, and performing end-to-end joint optimization training on it to obtain a relatively optimal optical coding structure and its corresponding computational decoding model; at the same time, the optical coding structure takes into account the quantization requirements in actual manufacturing during the optimization stage, so that the optimized diffractive optical element structure is consistent with the manufactured diffractive optical element structure, thereby improving the reconstructed image accuracy of the entire spectral imaging system. 7. A diffraction snapshot spectral imaging method as claimed in claim 6, characterized in that: the end-to-end joint optimization in step 104 can be implemented using any machine learning automatic differentiation framework; the end-to-end loss function used in the end-to-end joint optimization is

It is the L2 regularization loss part that includes the reconstructed image loss part, the adaptive quantization part and the decoding weight:

in,

is the spectral reconstruction loss function,

is the adaptive quantization loss, ω is the network weight of the reconstruction module, and β and γ are scaling parameters; Spectral reconstruction loss function

Reconstructed image

The mean absolute error MAE between the target spectral image I is calculated as:

Where K is the number of pixels in the image; Adaptive Quantization Loss

It is expressed as the mean square error MSE between the height map after quantization and the original height map:

Where J is the number of pixels in the height map. 8. A diffraction snapshot spectral imaging method as described in claim 7, characterized in that: a GPU is used to perform end-to-end joint optimization training on the joint encoding and decoding spectral imaging model constructed in step 104 to obtain a relatively optimal optical coding structure and its corresponding computational decoding model.

Images