WO2023279338A1

WO2023279338A1 - Neural spectral field reconstruction for spectrometer

Info

Publication number: WO2023279338A1
Application number: PCT/CN2021/105312
Authority: WO
Inventors: Zi WANG; Wei Lu; Jingyi Yu
Original assignee: Shanghaitech University
Priority date: 2021-07-08
Filing date: 2021-07-08
Publication date: 2023-01-12

Abstract

A neural spectral field (NeSF) reconstruction method for spectrometer is provided, comprising: encoding, by a computing system, a neural spectral field, wherein the neural spectral field comprises a neural network representing the spectrum as a function of a wavelength; and reconstructing the spectrum of the spectrometer by training the neural network using measured intensities of the spectrometer.

Description

NEURAL SPECTRAL FIELD RECONSTRUCTION FOR SPECTROMETER

TECHNICAL FIELD

The present invention relates to the field of spectral reconstruction, and in particular, to a neural spectral field reconstruction for spectrometer.

BACKGROUND

A spectrometer can be used to obtain the intensity of light reflected by a target at different wavelengths. It is a basic tool for spectral analysis and has important applications in spectral imaging, chemical industry, biopharmaceutical and other fields. There are many implementation principles of spectrometers, including narrowband filters, broadband filters, and Fourier spectrometers. The spectrometers implemented by these different methods can all be described by the same type of imaging model. However, how to reconstruct the target’s spectral curve from the known imaging model and measurement data remains a challenging problem. The amount of measurement data for reconstructing the spectrum is far less than the target data that needs to be acquired, and the existence of various noises makes the problem difficult to solve.

The traditional spectral reconstruction algorithms usually express the spectrum to be reconstructed as a vector variable and use the sparse characteristic of the spectrum in a specific transformation domain to recover the spectrum curve from the measurement result. However, simple sparse transforms such as discrete cosine transform and wavelet transform are generally performed on noisy underdetermined problems, while complex sparse transforms such as low-rank characteristics has high computational complexity and the reconstruction algorithm takes a long time.

The neural spectral field (NeSF) provided in accordance with the embodiments of the present invention starts from a completely different point of view, which uses neural networks to fit the function of the spectrum with wavelength and encodes its own structure as a priori into the representation of the spectrum. This way of expression does not require sparse transformation designed by hand, so it can be highly intelligent and automated. At the same time, because the neural spectral field only change the way of expressing the spectrum, it is possible to combine the neural spectral field with the traditional sparse transformation to improve the accuracy of spectral reconstruction.

The above information disclosed in this Background section is only for facilitating the understanding of the background of this invention, and may contain information that is not known to a person of ordinary skill in the art.

SUMMARY

In view of the limitations of existing technologies described above, the present invention provides a computer-implemented method for imaging a non-line-of-sight scene to address the aforementioned limitations. Additional features and advantages of this invention will become apparent from the following detailed descriptions.

One aspect of the present invention is directed to a computer-implemented method for reconstructing a spectrum of a spectrometer. The method may include: encoding, by a computing system, a neural spectral field, wherein the neural spectral field comprises a neural network representing the spectrum as a function of a wavelength; and reconstructing the spectrum of the spectrometer by training the neural network using measured intensities of the spectrometer.

In some embodiments, the method may further include: mapping the wavelength λ to

γ= (λ, sin (2 ⁰λ) , cos (2 ⁰λ) , sin (2 ¹λ) , cos (2 ¹λ) , …, sin (2 ^Nλ) , cos (2 ^Nλ) ) ^T;

and using the γ as the input of the neural network.

In some embodiments, the spectrometer may be characterized as:

wherein I _i, S, and D _i may be the measured intensity, spectrum, and forward model of the spectrometer, respectively.

In some embodiments, the spectrometer may comprise a filter, and may be characterized as:

wherein I _i, S, and D _i may be the measured intensity, spectrum, and response of the filter, respectively.

In some embodiments, the measured intensity of the spectrometer may be expressed as:

I=DS (λ, w)

wherein the parameter of the neural network may be denoted as w, the spectral curve may be denoted as S (λ, w) , and the response of the filter may be denoted as D.

In some embodiments, the spectrometer may be a Fourier spectrometer.

In some embodiments, the spectrometer may be a dispersive spectrometer.

In some embodiments, the reconstructing the spectrum of the spectrometer by training the neural network using measured intensities of the spectrometer may further comprise calculating the parameter w of the neural network iteratively through gradient descent to optimize the L2-norm in accordance with:

In some embodiments, the method may further include: adding a sparse prior to the reconstructed neural spectral field.

In some embodiments, the L2-norm may be optimized in accordance with:

wherein a sparse vector may be denoted as θ (λ, w) , a sparse transform may be denoted as Ψ: θ (λ, w) =ΨS (λ, w) , and a corresponding inverse transform may be denoted as Φ: S(λ, w) =Φθ (λ, w) .

Another aspect of the present invention is directed to a non-transitory storage medium of a computing system. The non-transitory storage medium may store instructions that, when executed by one or more processors of the computing system, cause the computing system to perform a method comprising: encoding, by a computing system, a neural spectral field, wherein the neural spectral field comprises a neural network representing the spectrum as a function of a wavelength; and reconstructing the spectrum of the spectrometer by training the neural network using measured intensities of the spectrometer.

and using the γ as the input of the neural network.

In some embodiments, the spectrometer may be characterized as:

I=DS (λ, w) wherein the parameter of the neural network may be denoted as w, the spectral curve may be denoted as S (λ, w) , and the response of the filter may be denoted as D.

In some embodiments, the spectrometer may be a Fourier spectrometer.

In some embodiments, the spectrometer may be a dispersive spectrometer.

In some embodiments, the L2-norm may be optimized in accordance with:

wherein a sparse vector may be denoted as θ (λ, w) , a sparse transform may be denoted as Ψ: θ (λ, w) =ΨS (λ, w) , and a corresponding inverse transform may be denoted as Φ:S(λ, w) =Φθ (λ, w) .

The foregoing general description and the following detailed description are merely examples and explanations and do not limit the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute a part of the description, illustrate embodiments consistent with this invention and, together with the description, serve to explain the disclosed principles. It is apparent that these drawings present only some embodiments of this invention and those of ordinary skill in the art may obtain drawings of other embodiments from them without exerting any creative effort.

FIG. 1 is a schematic diagram of working principle of a filter-based spectrometer in related art.

FIG. 2 is a schematic diagram of the structure of the multi-layer perception (MLP) used in neural spactral filed (NeSF) according to an embodiment of the present invention.

FIG. 3 is a schematic diagram of the NeSF reconstruction pipeline according to an embodiment of the present invention.

FIG. 4 is a schematic diagram of the reconstruction process of the NeSF according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as being limited to those set forth herein. Rather, these embodiments are presented to provide a full and thorough understanding of this invention and to fully convey the concepts of the exemplary embodiments to others skilled in the art.

In addition, the described features, structures, and characteristics may be combined in any suitable manner in one or more embodiments. In the following detailed description, many specific details are set forth to provide a more thorough understanding of this invention. However, those skilled in the art will recognize that the various embodiments can be practiced without one or more of the specific details or with other methods, components, materials, or the like. In some instances, well-known structures, materials, or operations are not shown or not described in detail to avoid obscuring aspects of the embodiments.

1. Data from Spectrometer

FIG. 1 is a schematic diagram of working principle of a filter-based spectrometer in related art. As shown in FIG. 1, a filter-based spectrometer may have multiple filters with different responses, and the corresponding detector may output the sum of the product of the spectrum to be measured and the response function in the full spectrum, which may be expressed as:

wherein I _i, S, and D _i are the measured intensity, spectrum, and response of the filter, respectively.

The continuous equation may be discretized as:

I=DS

wherein I∈R ^M, S∈R ^N, and D∈R ^M×N are the measured intensity, spectrum, and filter response, respectively. The spectrum S may be recovered from the known measured intensity I and filter response D.

2. Neural Spectral Representation

FIG. 2 is a schematic diagram of the structure of the multi-layer perception (MLP) used in neural spactral filed (NeSF) according to an embodiment of the present invention. As shown in FIG. 2, different from the existing methods where the vector S is directly used as a variable, the spectrum may be deemed as a function of the wavelength λ and can be fit using a neural network.

The neural network may be implemented as a multi-layer perceptron (MLP) , which consists of an input layer, many hidden layers and an output layer. The input layer may transform the input to neural nodes. Each hidden layer may consist of many neural nodes, which receive the weighted linear sum of previous layers and apply nonlinear activation such as relu () to feed into the next layer. The output layer may generate the spectral intensity according to the input of the neural network. In an embodiment of the present invention, the width, or the maximum number of nodes in each layer, of the neural network is 32, and the depth, or the number of the layers, of the neural network is 4.

FIG. 3 is a schematic diagram of the NeSF reconstruction pipeline according to an embodiment of the present invention. As shown in FIG. 3, the input of the neural network may be the wavelength λ, and the output may be the spectral intensity S (λ) at that wavelength. In this way, inputting different wavelengths into the neural network respectively, the spectral curve may be obtained. In the actual design of the network, in order to strengthen the neural network’s ability to express high-frequency signals, the wavelength λ may be mapped to

γ= (λ, sin (2 ⁰λ) , cos (2 ⁰λ) , sin (2 ¹λ) , cos (2 ¹λ) , …, sin (2 ^Nλ) , cos (2 ^Nλ) ) ^T,

and γ may be used as the input of the network.

3. Optimization

The spectral curve represented by the neural network without training is meaningless. Therefore, it is necessary to supervise and train the neural network using the spectral model and measurement data so that the neural network may accurately represent the spectral curve to be reconstructed.

The parameter of the neural network may be denoted as w, and the spectral curve expressed by the neural network may be denoted as S (λ, w) . For spectra with different wavelengths, the spectrum vector may be denoted as S (λ, w) ∈R ^N, and the measured intensity of the spectrometer may be expressed as:

I=DS (λ, w)

While the neural representation of spectrum vector is newly proposed in accordance with embodiments of the present invention, imaging models without neural representation have been widely analyzed by data scientists. Although noises exist in such imaging models, it is possible to overcome the noise effects by expressing the reconstruction process as an L2-norm optimization problem.

The reconstruction process may be expressed as an optimization problem of L2-norm as:

Then the parameter w of the neural network may be calculated iteratively through the gradient descent.

In addition, the sparse prior in the existing methods may be added to neural spectral field reconstruction. In particular, sparse transform may be denoted as Ψ: θ(λ, w) =ΨS (λ, w) and the corresponding inverse transform may be denoted as Φ: S(λ, w) =Φθ (λ, w) , where θ (λ, w) is the sparse vector. Then the optimization problem with sparse transformation prior may be expressed as:

Gradient descent and other methods may also be used to solve the optimization problems when training the network. After the network training is completed, the wavelength may be inputted into the network, and the spectral intensity at that wavelength may be outputted.

4. Results and Discussion

Some experimental measurements as well as reconstruction results from the proposed neural spectral field (NeSF) method are discussed below.

FIG. 4 is a schematic diagram of the reconstruction process of the NeSF according to an embodiment of the present invention. As shown in FIG. 4, the spectrum of an object acquired previously by the spectrometer and a set of spectral response of calibrated filters may be used to generate the measured intensity. The NeSF method is then used to obtain the reconstructed spectrum, which is compared with the ground truth to show the effectiveness of the NeSF method.

Although the reconstruction process in filtered-based spectrometers is mainly discussed herein, the NeSF methods discussed above are suitable for other spectrometers such as Fourier spectrometer and dispersive spectrometer. The main difference between different types of spectrometers is the forward model D. The NeSF methods provided in accordance with embodiments of the present invention do not have any requirements on the forward model D, and it may be applied to different spectrometers.

Claims

A computer-implemented method for reconstructing a spectrum of a spectrometer, comprising:

encoding, by a computing system, a neural spectral field, wherein the neural spectral field comprises a neural network representing the spectrum as a function of a wavelength; and

reconstructing the spectrum of the spectrometer by training the neural network using measured intensities of the spectrometer.
The method of claim 1, further comprising:

mapping the wavelength λ to

γ= (λ, sin (2 ⁰λ) , cos (2 ⁰λ) , sin (2 ¹λ) , cos (2 ¹λ) , …, sin (2 ^Nλ) , cos (2 ^Nλ) ) ^T;

and

using the γ as an input of the neural network.
The method of claim 1, wherein the spectrometer is characterized as:

wherein I _i, S, and D _i are measured intensity, spectrum, and a forward model of the spectrometer, respectively.
The method of claim 3, wherein the spectrometer comprises a filter, and is characterized as:

wherein I _i, S, and D _i are the measured intensity, spectrum, and response of the filter, respectively.
The method of claim 4, wherein the measured intensity of the spectrometer is expressed as:

I=DS (λ, w)

wherein a parameter of the neural network is denoted as w, spectral curve is denoted as S (λ, w) , and the response of the filter is denoted as D.
The method of claim 3, wherein the spectrometer is a Fourier spectrometer.
The method of claim 3, wherein the spectrometer is a dispersive spectrometer.
The method of claim 4, wherein the reconstructing the spectrum of the spectrometer by training the neural network using measured intensities of the spectrometer further comprises calculating the parameter w of the neural network iteratively through gradient descent to optimize the L2-norm in accordance with:
The method of claim 3, further comprises:

adding a sparse prior to the reconstructed neural spectral field.
The method of claim 9, wherein the L2-norm is optimized in accordance with:

wherein a sparse vector is denoted as θ (λ, w) , a sparse transform is denoted as Ψ: θ (λ, w) =ΨS (λ, w) , and a corresponding inverse transform is denoted as Φ: S (λ, w) =Φθ (λ, w) .
A non-transitory storage medium of a computing system storing instructions that, when executed by one or more processors of the computing system, cause the computing system to perform a method comprising:

encoding, by a computing system, a neural spectral field, wherein the neural spectral field comprises a neural network representing the spectrum as a function of a wavelength; and

reconstructing the spectrum of the spectrometer by training the neural network using measured intensities of the spectrometer.
The non-transitory storage medium of claim 11, wherein the method further comprises:

mapping the wavelength λ to

γ= (λ, sin (2 ⁰λ) , cos (2 ⁰λ) , sin (2 ¹λ) , cos (2 ¹λ) , …, sin (2 ^Nλ) , cos (2 ^Nλ) ) ^T;

and

using the γ as an input of the neural network.
The non-transitory storage medium of claim 11, wherein the spectrometer is characterized as:

wherein I _i, S, and D _i are measured intensity, spectrum, and a forward model of the spectrometer, respectively.
The non-transitory storage medium of claim 13, wherein the spectrometer comprises a filter, and is characterized as:

wherein I _i, S, and D _i are the measured intensity, spectrum, and response of the filter, respectively.
The non-transitory storage medium of claim 14, wherein the measured intensity of the spectrometer is expressed as:

I=DS (λ, w)

wherein a parameter of the neural network is denoted as w, spectral curve is denoted as S (λ, w) , and the response of the filter is denoted as D.
The non-transitory storage medium of claim 13, wherein the spectrometer is a Fourier spectrometer.
The non-transitory storage medium of claim 13, wherein the spectrometer is a dispersive spectrometer.
The non-transitory storage medium of claim 14, wherein the reconstructing the spectrum of the spectrometer by training the neural network using measured intensities of the spectrometer further comprises calculating the parameter w of the neural network iteratively through gradient descent to optimize the L2-norm in accordance with:
The non-transitory storage medium of claim 11, wherein the method further comprises:

adding a sparse prior to the reconstructed neural spectral field.
The non-transitory storage medium of claim 19, wherein the L2-norm is optimized in accordance with:

wherein a sparse vector is denoted as θ (λ, w) , a sparse transform is denoted as Ψ: θ (λ, w) =ΨS (λ, w) , and a corresponding inverse transform is denoted as Φ: S (λ, w) =Φθ (λ, w) .