CN114353946B - Diffraction snapshot spectrum imaging method - Google Patents
Diffraction snapshot spectrum imaging method Download PDFInfo
- Publication number
- CN114353946B CN114353946B CN202111635615.8A CN202111635615A CN114353946B CN 114353946 B CN114353946 B CN 114353946B CN 202111635615 A CN202111635615 A CN 202111635615A CN 114353946 B CN114353946 B CN 114353946B
- Authority
- CN
- China
- Prior art keywords
- diffraction
- optical element
- coding
- snapshot
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000001228 spectrum Methods 0.000 title claims abstract description 75
- 238000003384 imaging method Methods 0.000 title claims abstract description 67
- 230000003287 optical effect Effects 0.000 claims abstract description 141
- 238000005457 optimization Methods 0.000 claims abstract description 53
- 238000000034 method Methods 0.000 claims abstract description 50
- 238000004364 calculation method Methods 0.000 claims abstract description 27
- 238000013139 quantization Methods 0.000 claims abstract description 26
- 238000004519 manufacturing process Methods 0.000 claims abstract description 16
- 238000013528 artificial neural network Methods 0.000 claims abstract description 12
- 230000008569 process Effects 0.000 claims abstract description 7
- 238000010801 machine learning Methods 0.000 claims abstract description 4
- 238000000701 chemical imaging Methods 0.000 claims description 27
- 230000006870 function Effects 0.000 claims description 26
- 238000012549 training Methods 0.000 claims description 20
- 230000003595 spectral effect Effects 0.000 claims description 18
- 230000003044 adaptive effect Effects 0.000 claims description 13
- 238000012545 processing Methods 0.000 claims description 12
- 238000013461 design Methods 0.000 claims description 10
- 238000010586 diagram Methods 0.000 claims description 7
- 230000008447 perception Effects 0.000 claims description 6
- 230000010363 phase shift Effects 0.000 claims description 3
- 238000011002 quantification Methods 0.000 claims description 3
- 230000004044 response Effects 0.000 claims description 3
- 238000009966 trimming Methods 0.000 claims description 3
- 230000008901 benefit Effects 0.000 abstract description 5
- 238000005516 engineering process Methods 0.000 description 5
- 238000004422 calculation algorithm Methods 0.000 description 4
- 239000000463 material Substances 0.000 description 3
- 238000004088 simulation Methods 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 235000001466 Ribes nigrum Nutrition 0.000 description 2
- 241001312569 Ribes nigrum Species 0.000 description 2
- 238000012271 agricultural production Methods 0.000 description 2
- 230000004069 differentiation Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000002474 experimental method Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- HPTJABJPZMULFH-UHFFFAOYSA-N 12-[(Cyclohexylcarbamoyl)amino]dodecanoic acid Chemical compound OC(=O)CCCCCCCCCCCNC(=O)NC1CCCCC1 HPTJABJPZMULFH-UHFFFAOYSA-N 0.000 description 1
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- VYPSYNLAJGMNEJ-UHFFFAOYSA-N Silicium dioxide Chemical compound O=[Si]=O VYPSYNLAJGMNEJ-UHFFFAOYSA-N 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 230000000052 comparative effect Effects 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- PCHJSUWPFVWCPO-UHFFFAOYSA-N gold Chemical compound [Au] PCHJSUWPFVWCPO-UHFFFAOYSA-N 0.000 description 1
- 239000010931 gold Substances 0.000 description 1
- 229910052737 gold Inorganic materials 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- 238000010998 test method Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Investigating Or Analysing Materials By Optical Means (AREA)
- Spectrometry And Color Measurement (AREA)
Abstract
The invention discloses a diffraction snapshot spectrum imaging method, and belongs to the field of computational photography. The method is applied to a diffraction snapshot spectrum imaging system, and the coding structure of the diffraction optical element and the reconstruction decoding neural network are jointly optimized in a data driving mode to obtain a relatively optimal optical coding structure and a corresponding calculation decoding model, so that the coding and decoding parts are more matched with each other; the optical coding structure considers the quantization requirement in actual manufacture in the optimization stage, so that the diffraction optical element structure obtained by optimization is consistent with the manufactured diffraction optical element structure, and the accuracy of the reconstructed image is improved; the diffraction optical element with high coding freedom is adopted as a coding part, so that the device has the advantages of small volume and compact structure, and has the capability of real-time imaging. The whole coding and decoding process is modeled by adopting a differentiable model, so that the model can be realized and optimized in any machine learning automatic differential frame, and the universality of the invention is improved.
Description
Technical Field
The invention relates to a diffraction snapshot spectrum imaging method, in particular to a method capable of quickly acquiring high-quality hyperspectral images, and belongs to the field of computational photography.
Background
The hyperspectral imaging technology is a technology combining a space imaging technology and a spectrum imaging technology, can record space and spectrum information of an object at the same time, can obtain more abundant object details compared with the traditional imaging technology, and is widely applied to various fields such as agricultural production, material identification, geological exploration, biomedicine and the like. Conventional snapshot spectral imaging methods typically use multiple refractive or reflective lenses to achieve optical encoding and reconstruct the desired spectral image by way of computational decoding. The physical system corresponding to this method has a complex optical coding part, so that its volume is usually large. The diffraction optical element has high degree of freedom coding capability, and the diffraction snapshot spectral imaging method uses one diffraction optical element to replace a refractive lens or a reflective lens used in the previous system as an optical coding structure, so that the volume of the whole system is greatly reduced. However, existing diffraction spectrum imaging methods are usually heuristic for manually designing the structure of the diffractive optical element, such design methods require manual design or optimization solution using effective prior knowledge, and it is difficult to ensure that the final designed structure is a globally optimal solution. In addition, the structure of the diffraction optical element obtained by the existing design method is a high-precision nearly continuous structure, and the physical manufacturing process can only process diffraction optical elements with no more than 16 steps, so that the optimized structure and the actually manufactured structure are different, and further, a reconstruction algorithm designed together during optimization cannot reconstruct a spectrum image with high quality.
Disclosure of Invention
The method aims to solve the problems that the optical coding is complicated to optimize, and the structure of the optimized diffraction optical element is different from the structure of the actual physical manufacture in the existing diffraction snapshot spectrum imaging method. The invention discloses a diffraction snapshot spectrum imaging method which aims to solve the technical problems that: the coding structure of the diffraction optical element in the diffraction snapshot spectral imaging system and the reconstruction decoding neural network are jointly optimized in a data driving mode to obtain a relatively optimal optical coding structure, and meanwhile, the optical coding structure considers the quantization requirement in actual manufacturing in an optimizing stage, so that the structure of the diffraction optical element obtained by optimization is consistent with the structure of the manufactured diffraction optical element, and the accuracy of the reconstructed image of the whole spectral imaging system is further improved. In addition, the invention adopts the diffraction optical element with high coding freedom degree as the coding part, and the imaging system adopting the invention has the advantages of small volume and compact structure compared with the imaging system adopting various refraction or reflection lenses as the coding part.
In order to achieve the purpose, the invention adopts the following technical scheme.
The invention discloses a diffraction snapshot spectrum imaging method, which is applied to a diffraction snapshot spectrum imaging system, wherein an optical coding structure and a reconstruction algorithm of the diffraction snapshot spectrum imaging system are jointly optimized, and an optical coding structure most suitable for a spectrum imaging task and a corresponding calculation decoding model thereof are searched simultaneously in a data driving mode; modeling an optical physical coding part of a system by using a point spread function model, splicing a reconstruction decoder with the point spread function model to form a complete end-to-end model for joint optimization solution; meanwhile, in order to ensure that the diffraction optical element structure is quantized after optimization, the self-adaptive quantized perception training method is used for processing the height diagram of the diffraction optical element, so that the diffraction optical element structure obtained by optimization is consistent with the manufactured diffraction optical element structure, and the accuracy of the reconstructed image of the whole spectrum imaging system is further improved; after training, directly using the diffraction optical element design obtained by optimization to manufacture, and constructing a diffraction snapshot spectrum imaging real object system; in the built diffraction snapshot spectrum imaging physical system, a diffraction optical element carries out phase modulation on incident light, an RGB image sensor collects modulated light fields, and a trained convolutional neural network carries out reconstruction decoding on collected RGB images to obtain a finally required hyperspectral image. Under the advantage of keeping the small and compact volume of the diffraction snapshot spectrum imaging system, the invention performs joint optimization on the coding structure of the diffraction optical element and the reconstruction decoding neural network in the diffraction snapshot spectrum imaging system in a data driving mode to obtain a relatively optimal optical coding structure and a corresponding calculation decoding model thereof, thereby enabling the coding and decoding parts to be more matched with each other; meanwhile, the quantitative requirements in actual manufacturing are considered in the optical coding structure in the optimization stage, so that the structure of the diffraction optical element obtained by optimization is consistent with the structure of the diffraction optical element manufactured, the accuracy of the reconstructed image of the whole spectrum imaging system is further improved, the structure of the diffraction optical element obtained by optimization is consistent with the structure of the diffraction optical element manufactured, and the accuracy of spectrum image reconstruction is further improved. The invention is used for carrying a plurality of fields related to spectrum imaging, such as aerospace, geological survey, agricultural production, biomedicine and the like.
The invention discloses a diffraction snapshot spectrum imaging method, which comprises the following steps:
step 101: and constructing an optical physical coding forward model according to the diffraction optical theory.
The photophysical encoding forward model is built based on a point spread function model in step 101. According to the direction of incident light, when a wave field of a point light source wavelength lambda of a natural scene propagates a certain distance d and reaches a plane position (x, y) in front of a diffraction optical element of a diffraction optical encoder, the wave field is expressed as U 0 ：
Where i is a complex number unit. The diffractive optical element produces a phase modulation of the wave field, the modulated wave field U1 being denoted:
wherein a (x, y) is an aperture function, a value of 0 means preventing propagation of the position, and a value of 1 means allowing propagation of the position; phi (x, y, lambda) is the phase shift of the diffractive optical element at the planar position (x, y) for a wave field of wavelength lambda:
φ(x，y，λ)＝(n λ -1)H(x，y) (3)
wherein ,nλ Is the refractive index of the diffractive optical element for lambda wavelength light, and H (x, y) is the height of the diffractive optical element design structure at the planar position (x, y). The wavefield, after continuing z-distance travel, reaches the RGB image sensor plane, which is denoted as U 2 ：
wherein fx and fy The frequency variables corresponding to the spatial positions (x, y), respectively. The point spread function P is derived from the wavefield at the sensor plane, which is the square magnitude of the complex wavefield at the sensor plane:
P(x，y，λ)∝|U 2 (x，y，λ)| 2 (5)
after the point spread function is acquired, the convolution operation is performed on the target scene image I (x, y, λ, d) with the corresponding wavelength λ and the corresponding point spread function P (x, y, λ) respectively:
i.e. obtain modulated images I' (x, y, λ) of the corresponding wavelengths and scene depths; the obtained modulated scene I ' comprises modulated images I ' under each condition of the total set wavelength lambda and the total set depth D ' λ，d ：
I＝{I′ λ，d |λ∈Λ，d∈D} (7)
The modulated image passes through RGB image sensor according to the position [ lambda ] 0 ，λ 1 ]Spectral response curve R in the range c After acquisition, the observed modulated RGB image I is obtained c∈{R，G，B} (x，y)：
Where η is sensor noise. To this end, a forward model of the optical code is built as shown in equations (1-8).
Step 102: and (3) carrying out quantized perception processing on the diffraction optical element height map in the optical physical coding forward model constructed in the step (A) so as to enable the diffraction optical element height map to be in a quantized step pattern.
In step 102, the quantized sensing process is performed on the diffraction optical element height map, and the quantized sensed height map is shown as follows:
H aq ＝α×F(Q(H f ))+(1-α)×H f (9)
wherein ,Hf For the weight of the original height map with full precision, alpha is a quantized perception control parameter, and is changed along with the number of training steps, T 0 and T1 Training step numbers corresponding to the start and end of the quantized sensing respectively:
q is a quantization operation function, and the input height map H is quantized to be H max Pattern of L steps within:
f is an adaptive trimming function:
F(Q(H f ))＝Q(H f )+W l (12)
which dynamically adjusts W according to adaptive quantization loss l To adjust the physical height of each otherwise uniform step to an optimal quantized version.
Step 103: and constructing a calculation decoding model based on the differentiable neural network, and predicting and reasoning an original spectrum image reconstruction image by using the calculation decoding model.
Step 103, the computational decoding model is any differentiable neural network structure module N that modulates the image I from the optical encoding module c∈{R，G，B} (x, y) predictive reasoning of original spectral image reconstruction images
I.e. the computational decoding model is expressed as:
step 104: the method comprises the steps of performing joint optimization on a coding structure of a diffraction optical element in a diffraction snapshot spectrum imaging system and a reconstruction decoding neural network in a data driving mode to obtain a relatively optimal optical coding structure and a corresponding calculation decoding model thereof; meanwhile, in the optimization stage, the coding structure of the diffraction optical element is enabled to be in a quantization step mode through the quantized sensing processing in the step 102, so that the quantization requirement in actual manufacturing is considered in the optimization stage, the structure of the diffraction optical element obtained through optimization is enabled to be consistent with the structure of the manufactured diffraction optical element, and the accuracy of a reconstructed image of the whole spectrum imaging system is further improved.
The implementation method of step 104 is as follows: the learning rate, the weight initialization mode and the weight attenuation coefficient are respectively set for the optical physical coding forward model constructed in the step 101 and the calculation decoding model constructed in the step 103, and parameters of the batch processing size, the optimization method and the iteration times of joint optimization and the starting and ending of the quantized sensing processing in the step 102 are set. Taking the output of the optical physical coding forward model constructed in the step 101 as the input of the calculation decoding model constructed in the step 103, and performing end-to-end joint optimization training on the constructed joint coding and decoding spectral imaging model to obtain a relatively optimal optical coding structure and a corresponding calculation decoding model thereof; meanwhile, the optical coding structure considers the quantification requirement in actual manufacturing in the optimization stage, so that the diffraction optical element structure obtained by optimization is consistent with the manufactured diffraction optical element structure, and the accuracy of the reconstructed image of the whole spectrum imaging system is further improved.
The end-to-end joint optimization in step 104 can be implemented using any machine learning automatic differentiation framework. End-to-end joint optimization of end-to-end loss function
An L2 regularization loss portion for containing reconstructed image loss portion, adaptive quantization portion, and decoding weights:
wherein ,
is a spectral reconstruction loss function, +.>
Is an adaptive quantization loss, ω is the network weight of the reconstruction module, and β and γ are proportional parameters.
Spectral reconstruction loss function
From reconstructed image +.>
And the average absolute error MAE between the target spectrum image I is calculated to obtain:
where K is the number of pixels in the image.
Adaptive quantization loss
Expressed as the mean square error MSE between the height map after quantization operation and the original height map:
wherein J is the number of pixels in the height map;
further comprising step 105, step 106:
step 105: and (3) manufacturing a diffraction optical element according to the optical coding structure obtained by optimizing in the step (104), and building a diffraction snapshot spectrum imaging system by utilizing the diffraction optical element and the RGB image sensor, wherein the built diffraction snapshot spectrum imaging system can be reduced in volume due to high coding freedom degree of the diffraction optical element, and the compactness of the diffraction snapshot spectrum imaging system is improved.
Step 106: and (3) shooting a target scene by using the diffraction snapshot spectrum imaging system built in the step (105), and decoding and reconstructing an observation image shot by the physical system by using the calculation decoding model obtained by optimization in the step (104) to obtain a hyperspectral image corresponding to the required target scene, thereby improving the accuracy of the reconstructed image of the whole spectrum imaging system.
Preferably, the GPU is used to perform end-to-end joint optimization training on the joint codec spectrum imaging model constructed in step 104, so as to obtain a relatively optimal optical coding structure and a corresponding calculation decoding model thereof.
The beneficial effects are that:
1. according to the diffraction snapshot spectrum imaging method disclosed by the invention, the coding structure of the diffraction optical element in the diffraction snapshot spectrum imaging system and the reconstruction decoding neural network are jointly optimized in a data driving mode, so that the relatively optimal optical coding structure and the corresponding calculation decoding model are obtained, and further the coding and decoding parts are more matched with each other.
2. According to the diffraction snapshot spectrum imaging method disclosed by the invention, the quantification requirement in actual manufacturing is considered in the optimization stage of the optical coding structure, so that the structure of the diffraction optical element obtained by optimization is consistent with the structure of the diffraction optical element manufactured, and the accuracy of the reconstructed image of the whole spectrum imaging system is further improved.
3. The diffraction snapshot spectrum imaging method disclosed by the invention adopts the diffraction optical element with high coding freedom as the coding part, and compared with an imaging system which uses various refraction or reflection lenses as the coding part, the diffraction snapshot spectrum imaging method has the advantages of small volume and compact structure, can perform hyperspectral imaging without scanning in space or spectrum dimension, and has the capability of real-time imaging.
4. The diffraction snapshot spectrum imaging method disclosed by the invention adopts the differentiable model to model the whole encoding and decoding process, so that the model can be realized and optimized in any machine learning automatic differential frame, and the universality of the diffraction snapshot spectrum imaging method is improved.
Drawings
In order to more clearly illustrate the technical solution of the embodiments of the present invention, the following description will briefly explain the drawings that are required to be used in the embodiments.
FIG. 1 is a flow chart of a diffraction snapshot spectral imaging method provided by an embodiment of the present invention;
FIG. 2 is a schematic diagram of a physical system structure of a diffraction snapshot spectrum imaging method according to an embodiment of the present invention;
FIG. 3 is a diagram of a combined training framework of a diffraction snapshot spectral imaging method provided by an embodiment of the present invention;
FIG. 4 is a diagram illustrating a quantized perceptual process of a diffraction snapshot spectral imaging method according to an embodiment of the present invention;
FIG. 5 is a diagram of a reconstruction network structure of a diffraction snapshot spectral imaging method provided by an embodiment of the present invention;
in the figure: 101-diffractive optical element, 102-aperture, 103-clamp, 104-RGB image sensor.
Detailed Description
For a better description of the objects and advantages of the present invention, the following description of the invention refers to the accompanying drawings and examples.
Example 1:
as shown in fig. 2, an imaging system corresponding to the diffraction snapshot spectrum method disclosed in the present embodiment includes a diffractive optical element 101, an aperture 102, a holder 103, and an RGB image sensor 104. The aperture 102 is used to block unwanted light. The diffractive optical element 101 is used for modulating the incident light in phase. The clamp 103 is used to secure the diffractive optical element to the sensor. The RGB image sensor 104 is used to detect the modulated image.
Fig. 2 shows a schematic physical structure of an imaging system corresponding to the diffraction snapshot spectrum method provided by the embodiment of the invention, according to the outgoing direction of the light path, incident light sequentially passes through an aperture 102 and a diffraction optical element 101, is finally collected by an RGB image sensor 104, and is converted into an image digital signal, and the image digital signal is output to a spectrum reconstruction network and is subjected to image processing through the spectrum reconstruction network to realize spectrum imaging. The method for realizing spectral imaging by image processing through a spectral reconstruction network comprises the following steps: and (3) performing a linear demosaicing algorithm on the RAW image acquired by the RGB image sensor to obtain an RGB image, cutting the demosaiced RGB image into RGB images with the width and height of 2 and the size of n times, and inputting the RGB images into a pre-trained reconstruction network. Reconstructing an image output by the network to obtain a required spectrum image.
As shown in fig. 1, the diffraction snapshot spectrum imaging method disclosed in the embodiment includes the following steps:
step 101: and constructing an optical physical coding forward model according to the diffraction optical theory.
The photophysical encoding forward model is built based on a point spread function model in step 101. Firstly, according to the direction of incident light, when a wave field with wavelength lambda of a point light source of a natural scene is transmitted for a certain distance dWhen reaching the plane position (x, y) in front of the diffractive optical element of the diffractive optical encoder of the system according to the invention, it is denoted as U 0 ：
Wherein i is a complex unit, d is set to 1m, and the wavelength lambda sampling point comprises wavelengths of 400nm to 700nm at intervals of every 10 nm; the diffractive optical element generates a phase modulation of the wave field, the modulated wave field U 1 Expressed as:
wherein a (x, y) is an aperture function, a value of 0 means preventing propagation of the position, and a value of 1 means allowing propagation of the position; phi (x, y, lambda) is the phase shift of the diffractive optical element at the planar position (x, y) for a wave field of wavelength lambda:
φ(x，y，λ)＝(n λ -1)H(x，y) (3)
wherein ,nλ Is the refractive index of the diffraction optical element for lambda wavelength light, and the material of the embodiment is OHARA SK1300 quartz glass material; h (x, y) is the height of the diffractive optical element design structure at the planar location (x, y). The wavefield then continues to travel z-distance before reaching the RGB image sensor plane, which may be represented as U 2 ：
wherein fx and fy The frequency variables corresponding to the spatial positions (x, y) are respectively set to 50mm from the diffraction optical element to the RGB image sensor; the point spread function P is then derived, which is the square magnitude of the complex wave field on the sensor plane:
P(x，y，λ)∝|U 2 (x，y，λ)| 2 . (5)
after the point spread function is acquired, the convolution operation is performed on the target scene image I (x, y, λ, d) with the corresponding wavelength λ and the corresponding point spread function P (x, y, λ) respectively:
the modulated image I' (x, y, lambda) corresponding to the wavelength and the scene depth can be obtained; the finally obtained modulated scene I ' comprises modulated images I ' under each condition of the total set wavelength lambda and the total set depth D ' λ，d ：
I＝{I′ λ，d |λ∈Λ，d∈D} (7)
The modulated image passes through RGB image sensor according to the position [ lambda ] 0 ，λ 1 ]Spectral response curve R in the range c After acquisition, the observed modulated RGB image I is obtained c∈{R，G，B} (x，y)：
Where η is sensor noise, gaussian noise with a mean value of 0 and a standard deviation of 0.001 is set; the RGB sensor model adopted in the embodiment is FLIR GS3-U3-41S4C; to this end, a forward model of the optical code is built according to the above-mentioned relation. And the above model is implemented in the TensorFlow 2.
Step 102: and (3) carrying out quantized perception processing on the diffraction optical element height map in the optical physical coding forward model constructed in the step (A) so as to enable the diffraction optical element height map to be in a quantized step pattern.
In step 102, the quantized sensing process is performed on the diffraction optical element height map, and the quantized sensed height map may be expressed as:
H aq ＝α×F(Q(H f ))+(1-α)×H f (9)
wherein ,Hf For the weight of the original height map with full precision, alpha is a quantized perception control parameter, and is changed along with the number of training steps, T 0 and T1 Respectively set as 5 th trainingTraining steps corresponding to training turns and training steps corresponding to the 40 th training turn:
q is a quantization operation function, and the input height map H is quantized to be H max The pattern of L steps within, the number of steps L is 4:
f is an adaptive trimming function:
F(Q(H f ))＝Q(H f )+W l (12)
which dynamically adjusts W according to adaptive quantization loss l To adjust the physical height of each otherwise uniform step to an optimal quantized version.
Step 103: and constructing a calculation decoding model based on the differentiable neural network, and predicting and reasoning an original spectrum image reconstruction image by using the calculation decoding model.
The computational decoding model in step 103 is Res-UNet, denoted N, which is the image I modulated by the optical encoding module c∈{R，G，B} (x, y) predictive reasoning of original spectral image reconstruction images
I.e. can be expressed as:
step 104: the method comprises the steps of performing joint optimization on a coding structure of a diffraction optical element in a diffraction snapshot spectrum imaging system and a reconstruction decoding neural network in a data driving mode to obtain a relatively optimal optical coding structure and a corresponding calculation decoding model thereof; meanwhile, in the optimization stage, the coding structure of the diffraction optical element is enabled to be in a quantization step mode through the quantized sensing processing in the step 102, so that the quantization requirement in actual manufacturing is considered in the optimization stage, the structure of the diffraction optical element obtained through optimization is enabled to be consistent with the structure of the manufactured diffraction optical element, and the accuracy of a reconstructed image of the whole spectrum imaging system is further improved.
In step 104, the learning rates of the optical physical coding forward model and the calculation decoding model are initialized to 10-2 and 10-3 respectively, and the learning rate of the optical physical coding forward model and the calculation decoding model is reduced to 0.8 times of the original learning rate of the optical physical coding forward model and the calculation decoding model in each training 1 round on the training data set. Step 104, setting the batch size to 4, which represents the number of images processed by single iteration optimization; the optimization method is Adam, and the iteration number is 50 rounds; and sets the quantized perceptual training to begin at run 5 and end at run 40.
The end-to-end joint optimization in step 104 is implemented using the TensorFlow2 automatic differentiation framework, and the data set is an ICVL spectral scene data set. End-to-end joint optimization of end-to-end loss function
An L2 regularization loss portion for containing reconstructed image loss portion, adaptive quantization portion, and decoding weights:
wherein ,
is a spectral reconstruction loss function, +.>
Is self-adaptive quantization loss, ω is the network weight of the reconstruction module, and the ratio parameters β and γ are respectively 0.01 and 0.0001.
Spectral reconstruction loss function
From reconstructed image +.>
And the average absolute error MAE between the target spectrum image I is calculated to obtain:
where K is the number of pixels in the image.
Adaptive quantization loss
Expressed as the mean square error MSE between the height map after quantization operation and the original height map:
wherein J is the number of pixels in the height map;
step 105: and (3) manufacturing a diffraction optical element according to the optical coding structure obtained by optimizing in the step (104), and building a diffraction snapshot spectrum imaging system by utilizing the diffraction optical element and the RGB image sensor, wherein the built diffraction snapshot spectrum imaging system can be reduced in volume due to high coding freedom degree of the diffraction optical element, and the compactness of the diffraction snapshot spectrum imaging system is improved.
In step 105, physical parameters of the real object are consistent with all physical parameters of the simulation optimizing stage, namely, the distance from a shooting scene to a diffraction optical element is 1m, and the distance from the diffraction optical element to a sensor is 50mm.
Step 106: and shooting a target scene by using the diffraction snapshot spectrum imaging system built in the step 105, and decoding and reconstructing an observation image shot by the physical system by using the computing decoding model Res-UNet obtained by optimization in the step 104 to obtain a hyperspectral image corresponding to the required target scene, thereby improving the accuracy of the reconstructed image of the whole spectrum imaging system. Thus, diffraction snapshot spectral imaging is realized.
To illustrate the effects of the present invention, the present example will compare the joint optimization codec method including the Fresnel lens code CASSI code, the conventional joint optimization codec (Deep Optics), and the present example diffraction snapshot spectral imaging under the same experimental conditions.
1. Experimental conditions
The hardware test conditions of this experiment were: CPU adopts Intel (R) Xeon (R) Gold 5218R, running memory size is 256GB, GPU is NVIDIA GeForce RTX3090, GPU video memory is 24G, CUDA calculation library version is 11.0; the hyperspectral pictures used for the test are from an ICVL data set, and the input image size is 512 multiplied by 512; the test methods were all implemented using a TensorFlow 2.5. In order to ensure fairness of the comparison experiment, all physical parameters and reconstruction network structures of the models adopting different coding modes are consistent with those introduced by the embodiment.
2. Experimental results
To quantitatively measure the quality of the reconstructed results, the spatial quality and visual effect of the reconstructed results are measured using peak signal-to-noise ratio (Peak Signal to Noise Ratio, PSNR) and structural similarity (Structural Similarity, SSIM); the spectral fidelity of the reconstructed result is measured using root mean square error (Root Mean Square Error, RMSE) and relatively dimensionless global error (Relative Dimensionless Global Error in Synthesis, ERGAS);
table 1 results of index reconstruction from comparative experimental spectra of different snapshot spectral imaging methods
| Method\index | PSNR | SSIM | RMSE | ERGAS |
| Fresnel | 27.41 | 0.869 | 0.0556 | 32.18 |
| CASSI | 30.66 | 0.899 | 0.0354 | 20.74 |
| Deep Optics | 31.68 | 0.939 | 0.0319 | 19.16 |
| The embodiment of the invention discloses a method | 35.32 | 0.968 | 0.0205 | 12.35 |
Table 1 shows the spectral reconstruction index results on the test set after ICVL dataset training for the different snapshot spectral imaging methods. It can be seen that the method disclosed by the embodiment of the invention has better performance in both spatial image quality and spectral fidelity.
In summary, in the diffraction snapshot spectrum imaging method disclosed in the embodiment, the diffraction optical element is used as the optical coding part, so that the volume and complexity of the physical part are obviously reduced; meanwhile, the method for training in an automatic differential frame to jointly optimize and design the codec by using the constructed simulation model is based on a data driving mode, no additional priori knowledge is needed, the codec algorithms can be matched with each other without manual participation in design, and the method can generally find a globally optimal codec design scheme in a solution space; in addition, the diffraction snapshot spectrum imaging method provided by the embodiment also uses a quantized sensing method in the coding and decoding combined optimization stage to consider the quantized requirement in actual physical manufacturing, so that the difference between simulation and real objects is reduced, and a better spectrum reconstruction effect is achieved. The spectral imaging method provided by the embodiment has important application value in the low-light fields of remote sensing, biomedicine and the like and the high-speed measurement fields of dynamic object monitoring, tracking and the like.
The foregoing description is only of the preferred embodiments of the invention and is not intended to limit the invention thereto. The present invention may include various modifications and variations to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (8)
1. A diffraction snapshot spectral imaging method, characterized in that: the method comprises the following steps:
step 101: constructing an optical physical coding forward model according to a diffraction optical theory;
step 102: carrying out quantized sensing treatment on the diffraction optical element height diagram in the optical physical coding forward model constructed in the step one, so that the diffraction optical element height diagram presents a quantized step pattern;
step 103: constructing a calculation decoding model based on a differentiable neural network, and predicting and reasoning an original spectrum image reconstruction image by using the calculation decoding model;
step 104: the method comprises the steps of performing joint optimization on a coding structure of a diffraction optical element in a diffraction snapshot spectrum imaging system and a reconstruction decoding neural network in a data driving mode to obtain a relatively optimal optical coding structure and a corresponding calculation decoding model thereof; meanwhile, in the optimization stage, the coding structure of the diffraction optical element is enabled to be in a quantization step mode through the quantized sensing processing in the step 102, so that the quantization requirement in actual manufacturing is considered in the optimization stage, the structure of the diffraction optical element obtained through optimization is enabled to be consistent with the structure of the manufactured diffraction optical element, and the accuracy of a reconstructed image of the whole spectrum imaging system is further improved.
2. A diffraction snapshot spectral imaging method as set forth in claim 1, wherein: further comprising step 105, step 106:
step 105: manufacturing a diffraction optical element according to the optical coding structure obtained by optimization in the step 104, and building a diffraction snapshot spectrum imaging system by utilizing the diffraction optical element and the RGB image sensor, wherein the diffraction optical element has high coding freedom, so that the volume of the built diffraction snapshot spectrum imaging system can be reduced, and the compactness of the diffraction snapshot spectrum imaging system is improved;
step 106: and (3) shooting a target scene by using the diffraction snapshot spectrum imaging system built in the step (105), and decoding and reconstructing an observation image shot by the physical system by using the calculation decoding model obtained by optimization in the step (104) to obtain a hyperspectral image corresponding to the required target scene, thereby improving the accuracy of the reconstructed image of the whole spectrum imaging system.
3. A diffraction snapshot spectral imaging method according to claim 1 or 2, wherein: step 101, constructing the optical physical coding forward model based on a point spread function model; according to the direction of incident light, when a wave field of a point light source wavelength lambda of a natural scene propagates a certain distance d and reaches a plane position (x, y) in front of a diffraction optical element of a diffraction optical encoder, the wave field is expressed as U 0 ：
Wherein i is a complex number unit; the diffractive optical element generates a phase modulation of the wave field, the modulated wave field U 1 Expressed as:
wherein a (x, y) is an aperture function, a value of 0 means preventing propagation of the position, and a value of 1 means allowing propagation of the position; phi (x, y, lambda) is the phase shift of the diffractive optical element at the planar position (x, y) for a wave field of wavelength lambda:
φ(x，y，λ)＝(n λ -1)H(x，y) (3)
wherein ,nλ Is the refractive index of the diffractive optical element for lambda wavelength light, H (x, y) is the height of the diffractive optical element design structure at the planar position (x, y); the wavefield, after continuing z-distance travel, reaches the RGB image sensor plane, which is denoted as U 2 ：
wherein fx and fy Frequency variables corresponding to the spatial positions (x, y) respectively; the point spread function P is derived from the wavefield at the sensor plane, which is the square magnitude of the complex wavefield at the sensor plane:
P(x，y，λ)∝|U 2 (x，y，λ)| 2 (5)
after the point spread function is acquired, the convolution operation is performed on the target scene image I (x, y, λ, d) with the corresponding wavelength λ and the corresponding point spread function P (x, y, λ) respectively:
i.e. obtain modulated images I' (x, y, λ) of the corresponding wavelengths and scene depths; the obtained modulated scene I ' comprises modulated images I ' under each condition of the total set wavelength lambda and the total set depth D ' λ，d ：
I＝{I′ λ，d |λ∈Λ，d∈D} (7)
The modulated image passes through an RGB image sensor according to the modulated imageAt [ lambda ] 0 ，λ 1 ]Spectral response curve R in the range c After acquisition, the observed modulated RGB image I is obtained c∈{R，G，B} (x，y)：
Where η is sensor noise; to this end, a forward model of the optical code is built as shown in equations (1-8).
4. A diffraction snapshot spectral imaging method as recited in claim 3, wherein: in step 102, the quantized sensing process is performed on the diffraction optical element height map, and the quantized sensed height map is shown as follows:
H aq ＝α×F(Q(H f ))+(1-α)×H f (9)
wherein ,Hf For the weight of the original height map with full precision, alpha is a quantized perception control parameter, and is changed along with the number of training steps, T 0 and T1 Training step numbers corresponding to the start and end of the quantized sensing respectively:
q is a quantization operation function, and the input height map H is quantized to be H max Pattern of L steps within:
f is an adaptive trimming function:
F(Q(H f ))＝Q(H f )+W l (12)
which dynamically adjusts W according to adaptive quantization loss l To adjust the physical height of each otherwise uniform step to an optimal quantized version.
5. Such asA diffraction snapshot spectral imaging method as set forth in claim 4, wherein: step 103, the computational decoding model is any differentiable neural network structure module N that modulates the image I from the optical encoding module c∈{R，G，B} (x, y) predictive reasoning of original spectral image reconstruction images
I.e. the computational decoding model is expressed as:
6. a diffraction snapshot spectral imaging method as set forth in claim 5, wherein: the implementation method of step 104 is as follows: setting a learning rate, a weight initialization mode and a weight attenuation coefficient for the optical physical coding forward model constructed in the step 101 and the calculation decoding model constructed in the step 103 respectively, and setting parameters of batch processing size, an optimization method, iteration times and starting and ending of the quantized sensing processing in the step 102; taking the output of the optical physical coding forward model constructed in the step 101 as the input of the calculation decoding model constructed in the step 103, and performing end-to-end joint optimization training on the constructed joint coding and decoding spectral imaging model to obtain a relatively optimal optical coding structure and a corresponding calculation decoding model thereof; meanwhile, the optical coding structure considers the quantification requirement in actual manufacturing in the optimization stage, so that the diffraction optical element structure obtained by optimization is consistent with the manufactured diffraction optical element structure, and the accuracy of the reconstructed image of the whole spectrum imaging system is further improved.
7. A diffraction snapshot spectral imaging method as set forth in claim 6, wherein: the end-to-end joint optimization in step 104 can be implemented using any machine learning automatic differential framework; end-to-end joint optimization of end-to-end loss function
An L2 regularization loss portion for containing reconstructed image loss portion, adaptive quantization portion, and decoding weights:
wherein ,
is a spectral reconstruction loss function, +.>
Is self-adaptive quantization loss, omega is the network weight of the reconstruction module, and beta and gamma are proportional parameters;
spectral reconstruction loss function
From reconstructed image +.>
And the average absolute error MAE between the target spectrum image I is calculated to obtain:
wherein K is the number of pixels in the image;
adaptive quantization loss
Expressed as the mean square error MSE between the height map after quantization operation and the original height map:
where J is the number of pixels in the height map.
8. A diffraction snapshot spectral imaging method as set forth in claim 7, wherein: and (3) performing end-to-end joint optimization training on the joint coding and decoding spectral imaging model constructed in the step (104) by using the GPU to obtain a relatively optimal optical coding structure and a corresponding calculation and decoding model thereof.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111635615.8A CN114353946B (en) | 2021-12-29 | 2021-12-29 | Diffraction snapshot spectrum imaging method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111635615.8A CN114353946B (en) | 2021-12-29 | 2021-12-29 | Diffraction snapshot spectrum imaging method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114353946A CN114353946A (en) | 2022-04-15 |
| CN114353946B true CN114353946B (en) | 2023-05-05 |
Family
ID=81102545
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111635615.8A Active CN114353946B (en) | 2021-12-29 | 2021-12-29 | Diffraction snapshot spectrum imaging method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114353946B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115358381B (en) * | 2022-09-01 | 2024-05-31 | 清华大学 | Optical full adder and neural network design method, equipment and medium thereof |
| CN116704070B (en) * | 2023-08-07 | 2023-11-14 | 北京理工大学 | Method and system for reconstructing jointly optimized image |
| CN118135397A (en) * | 2024-03-01 | 2024-06-04 | 北京邮电大学 | Visible light hyperspectral compression acquisition chip parameter training and acquisition method and system based on diffraction coding |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109886898B (en) * | 2019-03-05 | 2020-10-02 | 北京理工大学 | Imaging method of spectral imaging system based on optimization heuristic neural network |
| CN109883548B (en) * | 2019-03-05 | 2020-04-21 | 北京理工大学 | Optimization heuristic-based coding optimization method for spectral imaging system of neural network |
| CN112985600B (en) * | 2021-02-04 | 2022-01-04 | 浙江大学 | Spectrum coding imaging system and method based on diffraction |
| CN113155284B (en) * | 2021-04-20 | 2022-07-26 | 浙江大学 | Refraction-diffraction mixed spectral coding imaging system and method |
-
2021
- 2021-12-29 CN CN202111635615.8A patent/CN114353946B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN114353946A (en) | 2022-04-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114353946B (en) | Diffraction snapshot spectrum imaging method | |
| US10274420B2 (en) | Compact multifunctional system for imaging spectroscopy | |
| CN109697697B (en) | Reconstruction method of spectral imaging system based on optimization heuristic neural network | |
| CN111047681B (en) | Single-pixel three-dimensional end-to-end reconstruction method and device based on deep learning | |
| CN109883548B (en) | Optimization heuristic-based coding optimization method for spectral imaging system of neural network | |
| CN109447891A (en) | A kind of high quality imaging method of the spectrum imaging system based on convolutional neural networks | |
| CN113762460B (en) | Multimode optical fiber transmission image migration reconstruction algorithm based on numerical value speckle | |
| CN109886898B (en) | Imaging method of spectral imaging system based on optimization heuristic neural network | |
| KR20190036442A (en) | Hyperspectral Imaging Reconstruction Method Using Artificial Intelligence and Apparatus Therefor | |
| CN115499566B (en) | End-to-end high quality achromatic imaging system based on depth calculation optical element | |
| CN111598962B (en) | Single-pixel imaging method and device based on matrix sketch analysis | |
| KR20190036459A (en) | Hyperspectral Imaging Reconstruction Method Using Artificial Intelligence and Apparatus Therefor | |
| Li et al. | Quantization-aware deep optics for diffractive snapshot hyperspectral imaging | |
| CN114897752A (en) | Single-lens large-depth-of-field calculation imaging system and method based on deep learning | |
| CN115512192A (en) | Multispectral and hyperspectral image fusion method based on cross-scale octave convolution network | |
| CN115731191A (en) | Narrow-band spectral imaging method based on neural network | |
| CN115439363A (en) | Video defogging device and method based on comparison learning | |
| Xu et al. | Hyperspectral image reconstruction based on the fusion of diffracted rotation blurred and clear images | |
| CN113256733B (en) | Camera spectral sensitivity reconstruction method based on confidence voting convolutional neural network | |
| CN109447890A (en) | A kind of Encoding Optimization of the spectrum imaging system based on convolutional neural networks | |
| Zhou et al. | RDFNet: regional dynamic FISTA-Net for spectral snapshot compressive imaging | |
| CN116433548A (en) | Hyperspectral and panchromatic image fusion method based on multistage information extraction | |
| CN115601611A (en) | Deep learning spectrum reconstruction method and system adaptive to exposure change | |
| CN114719980A (en) | End-to-end spectrum reconstruction method and system | |
| Wei et al. | Computational imaging-based single-lens imaging systems and performance evaluation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |