KR102132075B1 - Hyperspectral Imaging Reconstruction Method Using Artificial Intelligence and Apparatus Therefor - Google Patents

Abstract

Disclosed is a method and apparatus for reconstructing a hyperspectral image using artificial intelligence. An ultra-spectral image reconstruction method according to an embodiment of the present invention includes receiving coded data for an image; And reconstructing a hyperspectral image of the image for the coded data based on a pre-generated nonlinear learning model.

Description

Hyperspectral Imaging Reconstruction Method Using Artificial Intelligence and Apparatus Therefor}

The present invention relates to a hyperspectral image reconstruction technology, and in detail, artificial intelligence, for example, is a kind of convolutional neural network (CNN) and is of high quality using a nonlinear learning model, a convolutional autoencoder. It relates to a method and apparatus for reconstructing or reconstructing the hyperspectral image of the.

Unlike a conventional RGB camera, a hyperspectral image (or hyperspectral image) includes information through much higher density spectral sampling, that is, additional data. Here, additional data can be used for many applications, including exterior capture, environmental monitoring, scientific imaging, astronomy, and the like.

There are a number of studies focusing on developing a hardware architecture to capture hyperspectral information, the most direct approach being different bandpass or liquid crystal tunable flters (LCTF) while sequentially scanning visible spectral. ) Is a time-spectral scanning that isolates wavelength measurements. The spectral resolution of this approach is limited by the number of filters used. Time multiple sampling was introduced using a micro-translation stage or a digital micromirror device (DMD). In addition, the space-spectral scanning approach captures a sequence of images of each wavelength through slits. Other recent approaches include kaleidoscope-based multiple sampling, reconfigurable cameras, or designs to reduce hardware costs.

As mentioned above, existing solutions are focused on designing new hardware architectures, including the use of liquid crystal bandpass filters, pushhbroom scanners, micro conversion or digital mirror devices. However, existing solutions share limitations such as engineering costs and hardware deployment costs or the need to capture only static screens. Moreover, a common feature of existing solutions is the trade-off between spatial resolution and spectral accuracy in the captured results.

Several methods have been proposed to overcome this trade-off.

In order to emphasize the scarcity of the gradient, the optimization technique of a conventional embodiment undergoes a reconstruction process that defines a data accuracy term and a total standardized term for total variation, with the goal of overcoming the spatial-spectral trade-off. This optimization technique relies on two assumptions. First, the hyperspectral component exhibits a very high correlation in both spatial and spectral domains. Second, the hyperspectral vector belongs to the low-dimensional subspace. Techniques according to one conventional embodiment assume that a spectral uniform segment exists at the spectral level, and assume that the spectral gradient is roughly distinctly smooth, and introduce a limited optimization approach to infer spatial correlation. However, this approach still produces artificial results in reconstructed image structures and details.

Coded aperture snapshot spectral imaging (CASSI) of another conventional embodiment is one of the most widely used hyperspectral imaging methods capable of capturing dynamic scenes. By definition of this compression technique, the captured coding information must be reconstructed to produce the final image. CASSI has two classes depending on how the spectral signature is encoded: (1) spatially encoded CASSI (SD-CASSI) using a single spreader, and (2) spatial-spectral CASSI (coding for information in two regions). SS-CASSI) or double-disperser DDCASSI. All these techniques share an inherent trade-off between spatial resolution and spectral accuracy, so the reconstruction step determines the quality of the final image.

In the technique of another conventional embodiment, the data-based method learns a linear representation of a natural spectral image as a sparse coded dictionary, for example, an open hyperspectral image data set. In an example technique, a method for removing noise during reconstruction is proposed. Another example is the introduction of a dual camera system that combines high frame rate panchromatic video with low frame rate hyperspectral information. Panchromatic information can be used to learn overcomplete dictionaries. In another example technique, a space-spectral encoded hyperspectral imager equipped with a diffraction grating has been introduced.

Embodiments of the present invention, in detail, artificial intelligence, for example, a type of convolutional neural network (CNN) and a nonlinear learning model using a convolutional autoencoder, a high-quality hyperspectral image. Provided is a method and apparatus for reconfiguration.

An ultra-spectral image reconstruction method according to an embodiment of the present invention includes receiving coded data for an image; And reconstructing a hyperspectral image of the image for the coded data based on a pre-generated nonlinear learning model.

Furthermore, the hyperspectral image reconstruction method according to an embodiment of the present invention further includes generating a nonlinear learning model through training using a preset hyperspectral image data set, and the reconstructing the hyperspectral image comprises: The hyperspectral image may be reconstructed based on the generated nonlinear learning model and the nonlinear optimization technique.

The generating of the nonlinear learning model may generate the nonlinear learning model by learning a convolutional automatic encoder through learning using the hyperspectral image data set.

The generating of the nonlinear learning model may generate the nonlinear learning model using a decoder network after training the encoder network in a nonlinear space using the convolutional automatic encoder.

The automatic convolution encoder may include an encoder network that converts the hyperspectral image data set into a nonlinear representation and a decoder network that generates an original data set from the nonlinear representation.

The nonlinear learning model may be a learning model that outputs a nonlinear reconstruction of hyperspectral images.

Reconstructing the hyperspectral image is based on the nonlinear optimization technique that jointly normalizes the accuracy of the nonlinear spectral representation for the generated nonlinear learning model and the sparsity of gradients in the spatial domain. It is possible to reconstruct the hyperspectral image of the image for the coded data.

The reconstructing of the hyperspectral image may reconstruct the hyperspectral image of the image with respect to the coded data by repeatedly performing the nonlinear optimization technique using an alternating direction multiplier (ADMM) method.

The receiving step may receive coded data for the image using a compressed hyperspectral imaging approach.

An ultra-spectral image reconstruction method according to another embodiment of the present invention comprises the steps of receiving coded data for the image; And reconstructing a hyperspectral image of the image for the coded data based on the previously learned spectral fryer.

Furthermore, a method for reconstructing a hyperspectral image according to another embodiment of the present invention includes learning the spectral prior using a preset hyperspectral image data set, and reconstructing the hyperspectral image. May reconstruct the hyperspectral image based on the learned spectral fryer and the nonlinear optimization technique.

Reconstructing the hyperspectral image is based on the nonlinear optimization technique that jointly normalizes the accuracy of the nonlinear spectral representation for the learned spectral fryer and the sparsity of gradients in the spatial domain. It is possible to reconstruct the hyperspectral image of the image for the coded data.

The reconstructing of the hyperspectral image may reconstruct the hyperspectral image of the image with respect to the coded data by repeatedly performing the nonlinear optimization technique using an alternating direction multiplier (ADMM) method.

An apparatus for reconstructing hyperspectral images according to an embodiment of the present invention includes a receiver configured to receive coded data for an image; And a reconstruction unit reconstructing a hyperspectral image of the image for the coded data based on a previously generated nonlinear learning model.

Furthermore, the apparatus for reconstructing hyperspectral images according to an embodiment of the present invention further includes a generation unit for generating a nonlinear learning model through learning using a preset hyperspectral image data set, and the reconstruction unit is configured to generate the nonlinear learning model. The hyperspectral image may be reconstructed based on a nonlinear optimization technique.

The generation unit may generate the nonlinear learning model by learning a convolutional automatic encoder through learning using the hyperspectral image data set.

The generation unit may train the encoder network in a nonlinear space using the convolutional automatic encoder and then generate the nonlinear learning model using a decoder network.

The automatic convolution encoder may include an encoder network that converts the hyperspectral image data set into a nonlinear representation and a decoder network that generates an original data set from the nonlinear representation.

The nonlinear learning model may be a learning model that outputs a nonlinear reconstruction of hyperspectral images.

The reconstruction unit is an image for the coded data based on the nonlinear optimization technique that jointly normalizes the accuracy of nonlinear spectral representation for the generated nonlinear learning model and the sparity of gradients in the spatial domain. Can reconstruct the hyperspectral image of.

The reconstruction unit may reconstruct the hyperspectral image of the image for the coded data by repeatedly performing the nonlinear optimization technique using an alternating direction multiplier (ADMM) method.

The receiver can receive coded data for the image using a compressed hyperspectral imaging approach.

A method of reconstructing a hyperspectral image according to another embodiment of the present invention includes receiving coded data for an image; And reconstructing a hyperspectral image of the image for the coded data based on a pre-trained learning model and optimization technique.

According to embodiments of the present invention, artificial intelligence, for example, is a kind of convolutional neural network (CNN) and reconstructs high-quality hyperspectral images using a nonlinear learning model, a convolutional autoencoder. By doing so, it can be applied to any existing compression imaging architecture.

According to embodiments of the present invention, in terms of spectral accuracy and spatial resolution, the computational complexity can be reduced by a factor of two with respect to the sparse coding technique while surpassing the state-of-the-art method of each architecture.

According to embodiments of the present invention, it is possible to provide a new high-resolution hyperspectral data set that includes a clear image of more spectral than the existing method provided through the project website.

-충실도 프라이어의 영향에 대한 일 예시도를 나타낸 것이다.
도 10은 숨겨진 레이어 수가 재구성의 공간 해상도에 미치는 영향을 나타낸 것이다.
도 11은 본 발명과 기존 방법들에 대한 재구성 결과를 나타낸 것이다.
도 12는 본 발명에 의해 생성된 초분광 영상 데이터 세트에 대한 예시도를 나타낸 것이다.
도 13은 본 발명과 실측값 그리고 다른 방법들과 비교한 결과를 나타낸 것이다.
도 14는 본 발명과 회귀 기반 방법에 대해 스펙트럴 반사 정확도와 공간 구조 정확도를 비교한 일 예시도를 나타낸 것이다.
도 15는 본 발명의 초분광 영상 시스템과 이에 대한 결과의 예시도를 나타낸 것이다.
도 16은 본 발명이 스펙트럴 재구성에 대한 결과를 나타낸 것이다.
도 17은 본 발명에서의 제한 예를 나타낸 것이다.1 shows an exemplary diagram comparing a method according to the present invention and an existing method.
2 shows an exemplary diagram for spatial encoding used in SD-CASSI, SS-CASSI, and dual spreader DD-CASSI systems.
3 shows an overview of a two-step process for reconstructing a hyperspectral image from an encoded sensor signal.
4 shows an exemplary diagram for a convolutional automatic encoder.
5 shows an exemplary diagram for the reconstruction accuracy of an automatic encoder.
6 shows an exemplary diagram for an average result for a Columbia image data set.
7 shows a contrast comparison for a color checker hyper spectral image of the Columbia data set.
8 shows the results of reconstructing a standard spatial frequency measurement chart (ISO 12233) using Twist, SpaRSA, sparse coding, and the method of the present invention.
9 is the present invention

-Fidelity is an example of the effect of the fryer.
10 shows the effect of the number of hidden layers on the spatial resolution of reconstruction.
11 shows the reconstruction results for the present invention and the existing methods.
12 shows an exemplary view of a hyperspectral image data set generated by the present invention.
13 shows the results of comparing the present invention with actual values and other methods.
14 shows an exemplary diagram comparing spectral reflection accuracy and spatial structure accuracy for the present invention and the regression-based method.
15 shows an exemplary view of the hyperspectral imaging system of the present invention and its results.
Figure 16 shows the results of the present invention for spectral reconstruction.
17 shows an example of limitation in the present invention.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited or limited by the embodiments. In addition, the same reference numerals shown in each drawing denote the same members.

The present invention replaces the hand-crafted made fryer with a data-driven prior trained with a neural network, thereby reducing side effects caused by problems and introducing nonlinear representation of natural hyperspectral images. It can be enabled.

The novel spectral reconstruction algorithm of the present invention can be applied to the input captured by any compression imaging technology such as SD-CASSI, SS-CASSI and DD-CASSI, and is faster and better than other conventional data driving methods. Provides

The present invention uses a convolutional automatic encoder that generates a nonlinear representation. The present invention can improve accuracy and efficiency by jointly adjusting the accuracy of the nonlinear representation through a new global optimization technique.

The present invention proposes a new hyperspectral image reconstruction algorithm that overcomes the old trade-off between spectral accuracy and spatial resolution in conventional compression imaging techniques.

The method according to the invention consists of two steps. First, the present invention learns a nonlinear spectral representation from an actual hyperspectral (or hyperspectral) data set. To this end, we build a convolutional automatic encoder that can reconstruct its own input through an encoder and decoder network. Next, the present invention formulates a new optimization method that jointly normalizes the accuracy of nonlinear spectral representations learned by a new accuracy prior and the sparsity of gradients (or gradients) in the spatial domain. do. The method according to the present invention can be applied to any existing compression imaging architecture, and the results can be confirmed through tests through simulation and prototype hyperspectral imaging system construction. The technology according to the present invention surpasses the state-of-the-art method of each architecture in terms of spectral accuracy and spatial resolution, while the computational complexity can be reduced by a factor of two with respect to sparse coding technology. In addition, the present invention can provide hyperspectral interpolation and demosaicing (converting a sample made by a mosaic type color filter arrangement of a digital camera into a full color image), which is provided through a project website. It can provide a new high-resolution hyperspectral data set that contains sharper images of more spectral than conventional methods.

In addition, the present invention provides in-depth analysis of all factors and parameters of the reconstruction algorithm, and can capture new high-resolution hyperspectral image data sets. Here, the data set may be a modification of a limited portion of another existing data set in which the image is slightly out of focus due to low spatial resolution or exhibits a limited spectral range.

The present invention may make the new database publicly available on the project website along with the model and code of the present invention.

The present invention is characterized by using artificial intelligence, for example, a convolutional neural network (CNN) and a nonlinear learning model, a convolutional autoencoder, and a nonlinear optimization technique jointly combined. Is done. The present invention will be described in detail with reference to FIGS. 1 to 17.

The spectral signature imprinted by the coded aperture is the basic building block of a compressed hyperspectral image. From this, the image is reconstructed by means of optimization. The two main ways to encode this spectral information are spatial encoding and spatial-spectral encoding. FIG. 2 shows an exemplary diagram for spatial encoding used in the SD-CASSI, SS-CASSI and dual spreader DD-CASSI systems, as shown in spatial encoding used in the SD-CASSI system shown in FIG. 2A. The spectral coded projection is first generated and then sheared by dispersion. Therefore, the reconstruction step of SD-CASSI reconstructs the image with the sheared and coded information. And, as shown in the spatial encoding used in the SS-CASSI and double-disperser DD-CASSI systems shown in Fig. 2B, the approach first disperses the incident light beam, then creates a mask-coded projection, and the additional optical device Correct the information. As a result, spectral reconstruction of SS-CASSI and DD-CASSI requires simpler optimization than SD-CASSI, which provides excellent results at a more complex optical installation cost.

The present invention focuses on reconstructing hyperspectral images using compressed input. Thus, the method according to the invention is agnostic for a particular encoding of input spectral data. Considering the computational advantage, the present invention uses spatial spectral encoding as the first option for testing the reconstruction algorithm, and a detailed description thereof will be described later.

Image formation

는 위치

에서 파장

를 갖는 광의 분광 강도를 나타낸다. 마스크는 분산 함수

에 따라, 분산이 전송 함수

에 의해 주어진 코딩된 패턴을 생성하고, 분산은 수평 축을 따라 전단을 생성한다. 공간 인코딩 예를 들어, SD-CASSI에서 센서

상의 투사된 광 세기는 모든 가시 파장

에 걸쳐 적분으로서 아래 <수학식 1>과 같이 나타낼 수 있다.

Is located

Wavelength

It represents the spectral intensity of light having a. Mask is a variance function

Depending on the variance, the transfer function

Generates the coded pattern given by, and variance produces shear along the horizontal axis. Spatial encoding e.g. from SD-CASSI sensor

The projected light intensity of the image is all visible wavelengths.

The integral can be expressed as <Equation 1> below.

[Equation 1]

는 코딩된 마스크

에 의해 변조되고, 결과는

에 의해 반대 방향으로 비시각화되며, 센서

상의 투사된 광 세기는 모든 가시 파장

에 걸쳐 적분으로서 아래 <수학식 2>와 같이 나타낼 수 있다.In contrast, the shear spectrum in the case of space-spectral encoding e.g. DD-CASSI

Is a coded mask

Modulated by, and the result is

Is visualized in the opposite direction by the sensor

The projected light intensity of the image is all visible wavelengths

The integral can be expressed as <Equation 2> below.

[Equation 2]

의 부호는 역전될 수 있다.Where the horizontal variance function

The sign of can be reversed.

채널을 갖는 초분광 영상(또는 하이퍼스펙트럴 이미지)는

로 표현 될 수 있는데, 여기서

와 H와 W는 이미지의 공간 차원일 수 있다. 투과율은 희소 변조 행렬

를 통해 나타낼 수 있는데, 여기서

는 센서의 픽셀 수 일 수 있다. 이 행렬은 각 파장에 대한

부분 행렬로 구성될 수 있으며,

와

의 결과는 아래 <수학식 3>에 나타낸 바와 같이 캡쳐된 이미지

을 도출한다.In matrix vector form,

Hyperspectral images (or hyperspectral images) with channels

Can be expressed as, where

And H and W may be spatial dimensions of the image. Transmittance is sparse modulation matrix

Can be represented by, where

May be the number of pixels of the sensor. This matrix is for each wavelength

It can be composed of partial matrices,

Wow

The result of the image captured as shown in <Equation 3> below

To derive.

[Equation 3]

이므로 상당히 불확실한 시스템을 설명한다.This equation

Therefore, it describes a fairly uncertain system.

The present invention is meaningful in that it uses a nonlinear operator trained through a convolutional auto-encoder instead of a common sparse coding approach that uses a linear combination of overcomplete dictionaries.

Normal compression detection vs. compression Hyperspectral Imaging

Compression hyperspectral imaging (HSI) can be considered in terms of the approach of general compression sensing (CS). However, compressed HSI has several characteristics that require a more specialized solution to achieve high quality results. CS reconstructs the spatial image structure in the 2D patch. The color information is implicitly reconstructed by combining the reconstruction of three separately calculated color channels. Conversely, HSI reconstructs the spectral image into a 3D tensor and has higher complexity by performing stronger compression at the spectral level. Since it appears to overlap due to dispersion, it is not possible to reconstruct the color as in normal CS.

In the HSI approach of the present invention, a monochromatic sensor captures 31 spectral channels. As shown in the upper left of FIG. 1, the variance combining spectral and spatial domains (which is clearly seen in the captured coding information) leads to a general tradeoff between spatial and spectral resolution in HSI.

As shown in Fig. 1, the hyperspectral reconstruction algorithm according to the present invention works in conjunction with the input of other conventional compression imaging architectures and can produce high quality results. These results appear both in terms of spectral accuracy and spatial resolution, and according to the comparison results, it can be seen that the results of the present invention (Ours) are significantly improved compared to other conventional methods. For example, TwIST and SpaRSA generally provide sub-optimal spatial reconstruction, but sparse coding can provide color chart noise reconstruction. However, sparse coding does not accurately reconstruct the green rim on the coffee cup. The chart on the right shows how well the reconstruction of the present invention fits the actual data. Furthermore, the present invention can provide a new high resolution hyperspectral image data set.

Hyperspectral Image reconstruction ( HYPERSPECTRAL IMAGE RECONSTRUCTION)

3 shows an overview of a two-step process for reconstructing a hyperspectral image from an encoded sensor signal. First, we train an automatic convolutional encoder to learn the nonlinear representation of a real hyperspectral image tensor. This nonlinearity is a key aspect of the reconfiguration scheme of the present invention as it allows the present invention to cover a wider range of actual spectral functions. Second, the present invention reconstructs the hyperspectral image from the encoded input by solving the nonlinear optimization problem globally. As an important aspect of the present invention, a new fryer forcing the actual spectrum data-based automatic encoding representation into a reconstructed signal is introduced. The formula of the present invention is to reconstruct the final hyperspectral image by jointly adjusting the sparsity of the term and the gradient.

Convolutional neural networks can be used to classify spectral images, or to extract features from images. However, the present invention is not intended to reconstruct the original signal from the extracted characteristics. On the other hand, the automatic encoding device may be a neural network in which the output layer and the input layer share the same number of nodes and can reconstruct their own input through the encoder and decoder functions, which can be used for spectral image classification or denoising. . In the existing example technique, an automatic convolution encoder is proposed, where the convolution operation and the activation function operate at each layer. It is successfully applied to object retrieval, image classification, de-noising, or real-time correction of multipath interference in real-time flight images.

In the present invention, an encoder network is first trained to learn the representation of a hyperspectral image in a nonlinear space by utilizing a convolutional automatic encoder, and then a final image is reconstructed from coded sensor data using a decoder network.

Similar to the existing technique, the present invention assumes that the hyperspectral vector belongs to the subspace of the hidden expression. However, the present invention uses an automatic convolutional encoder to decompose the input signal into a base vector and a set of coefficients, instead of using a predetermined base (such as discrete cosine transform or wavelet) or dictionary-based sparse coding. . Moreover, a common sparse coding approach generally reconstructs the signal with a linear combination of basic functions, but automatic encoding allows nonlinear reconstruction of hyperspectral information to better meet the nonlinearity of the problem, leading to better results.

The convolutional automatic encoder in the present invention can be composed of two sub-networks, as shown in Fig. 4, from an encoder network (green block in Fig. 4) that converts an input learning data set into a non-linear representation and from this non-linear representation. It is the decoder network (red block in Fig. 4) that creates the original data set. Formally, the convolutional automatic encoder A() is a synthesis of the encoder function E() and decoder function D().

를 사용하여 비선형 표현

로 변환할 수 있다. 그러면 디코더 함수

를 사용하여

에서 하이퍼스펙트럴 이미지 프라이어로 사용되는 h를 재구성 할 수 있으며, 아래 <수학식 4>와 같이 나타낼 수 있다.In the present invention, after learning the network, the hyperspectral image h is an encoder function.

Nonlinear representation

Can be converted to Decoder function

use with

H can be reconstructed as a hyperspectral image fryer, and can be expressed as <Equation 4> below.

[Equation 4]

In the signal reconstruction process of the present invention, a nonlinear hyperspectral expression satisfying the image forming model of the present invention can be searched.

Network architecture

레이어로 구성될 수 있다.As shown in Fig. 4, the automatic encoding program of the present invention excludes the input and output layers.

It can be composed of layers.

는 각 하위 네트워크의 숨겨진 레이어 수를 의미할 수 있다.here,

Can mean the number of hidden layers of each sub-network.

는 자동 인코딩 장치의 시작 부분에 배치되며,

의 입력이 인코더 네트워크에 공급된다고 가정할 때 인코더는 아래 <수학식 5>, <수학식 6>와 같이 정의된 하이퍼스펙트럴 이미지의 비선형 표현을 출력한다.Encoder network

Is placed at the beginning of the automatic encoding device,

Assuming that the input of is supplied to the encoder network, the encoder outputs a nonlinear representation of the hyperspectral image defined as <Equation 5> and <Equation 6> below.

[Equation 5]

[Equation 6]

및

는 각각 커널 가중치, 중간 특징 표현 및 인코더 네트워크의 계층 l에서의 바이어스를 의미하며, 가중치와 바이어스는 자동 인코딩을 구성하고, 아래 첨자 E는 인코더를 나타낼 수 있으며,

는 직선 선형 단위(ReLU) 인 비선형 활성화 함수를 의미할 수 있다.here,

And

Denotes kernel weight, intermediate feature representation, and bias in layer l of the encoder network, respectively, weight and bias constitute automatic encoding, and subscript E can represent the encoder,

May denote a nonlinear activation function that is a linear linear unit (ReLU).

를 입력 하이퍼스펙트럴 이미지 h로 설정한다. 재구성 단계에서

에 제약을 가하지 않기 위해서, 활성화 함수는 상기 수학식 5의 출력 계층에 적용되지 않을 수 있다.

Set to the input hyperspectral image h. In the reconstruction phase

In order not to impose a constraint on, the activation function may not be applied to the output layer of Equation (5).

Similar to the architecture of the encoder network, a decoder network having a hidden layer can be defined as <Equation 7> and <Equation 8> below.

[Equation 7]

[Equation 8]

는 하이퍼스펙트럴 이미지의 비선형 표현

를 의미할 수 있다.here

Is a non-linear representation of a hyperspectral image

Can mean

에 대해

를 사용하지만 다른 계층은

의 커널로 컨볼루션된다. 이미지와 숨겨진 레이어의 공간 해상도는 동일하게 유지된다. 본 발명은 컨볼루션 자동 인코딩 장치의 기존 응용 프로그램과 같은 특징 벡터를 추출하는 대신 원본 신호를 재구성하는 것이다. 이러한 의미에서 본 발명의 특징 벡터는 하이퍼스펙트럴 벡터를 정의하는 저차원 부분 공간으로 볼 수 있으며 희소 코딩에서의 과도 완료된(overcomplete) 사전과 유사하다. 본 발명은 도 5a에 도시된 바와 같이, 더 많은 특징 벡터에 의해 더 많은 결과를 산출하는 것을 통해 벡터

의 개수가 재구성된 신호의 정확도에 중요한 영향을 미치는 것을 알 수 있다. 그러나 성능과 메모리 사이에는 실질적인 트레이드 오프가 있다. 실제로는 각 계층의 특징 벡터 수를

(원래 C = 31보다 큼)로 고정한다. 도 5b와 도 5c는 R = 64 개의 특징 벡터와 11개의 숨겨진 레이어를 갖는 하이퍼스펙트럴 재구성의 예를 보여준 것으로, 재구성된 스펙트럴 정보는 사실상 원본과 동일하다.The first convolutional layer of the encoder network,

About

But the other layers

Is convolved into the kernel. The spatial resolution of the image and the hidden layer remains the same. The present invention is to reconstruct the original signal instead of extracting a feature vector like the existing application of the convolutional automatic encoding device. In this sense, the feature vector of the present invention can be viewed as a low-dimensional subspace defining a hyperspectral vector and is similar to an overcomplete dictionary in sparse coding. The present invention, as shown in Figure 5a, through the production of more results by more feature vectors, vectors

It can be seen that the number of has a significant effect on the accuracy of the reconstructed signal. However, there is a real trade-off between performance and memory. The number of feature vectors in each layer

(Original C = greater than 31). 5B and 5C show an example of hyperspectral reconstruction having R = 64 feature vectors and 11 hidden layers, and the reconstructed spectral information is substantially the same as the original.

training step

를 포함한다. 하이퍼스펙트럴 이미지의 비선형 표현을 배우기 위해 자동 인코딩 네트워크를 학습시키고 손실 함수를 최소화하는 특정 세트

를 찾는다.

하이퍼스펙트럴 이미지의 집합

에 대해 overfitting을 피하기 위해 decay term을 포함하는 손실 함수

는 아래 <수학식 9>와 같이 나타낼 수 있다.The definition of the invention for an automatic encoder is a set of parameters

It includes. A specific set of training automatic encoding networks to minimize nonlinear representation of hyperspectral images and minimizing loss functions

Find

Set of hyperspectral images

Loss function with decay term to avoid overfitting

Can be expressed as <Equation 9> below.

[Equation 9]

는 대안 적으로

로 표현 될 수 있고,

는 오버피팅을 피하기 위해 데이터 정확도와 정규화 사이의 상대적 중요성을 조정할 수 있다. here,

Alternatively

Can be expressed as,

Can adjust the relative importance between data accuracy and normalization to avoid overfitting.

와

를 모두 초기화한다.The present invention uses a normalized initialization to maintain changes in back propagation gradients and activation.

Wow

Initialize all.

Implementation details

The present invention can create augmented training data sets using 109 hyperspectral images from Harvard and Columbia (77 images from Harvard and 32 from Columbia), which are publicly available. Each hyperspectral image contains about 31 wavelength channels.

The present invention can add an initial image data set according to an existing network learning approach. To achieve scale invariance for the input image, the input data set can be extended to two additional resolutions (1/2 and 2x), thus creating 327 hyperspectral images. In this increased data set, 21,760 tensor patches of size 96 x 96 x 31 are sampled.

인 64로 설정될 수 있으며, 감쇠 항의 가중치

는

로 설정될 수 있다. R = 64개의 특징 채널과 11개의 숨겨진 레이어를 사용하여 64GB의 메모리가 있는 i7-6770k CPU와 12GB의 메모리가 있는 NVIDIA Titan X Pascal GPU가 장착된 컴퓨터를 사용하여 네트워크를 학습하는데 약 30시간이 소요될 수 있다.The present invention implements an automatic encoder using TensorFlow and minimizes the loss function of Equation 9 using the ADAM gradient descent method, and can train up to 60 epochs. For gradient descent, the batch size has a faster learning speed.

Can be set to 64, the weight of the damping term

The

Can be set to R = It takes about 30 hours to train the network using a computer with an i7-6770k CPU with 64 GB of memory and an NVIDIA Titan X Pascal GPU with 12 GB of memory using 64 feature channels and 11 hidden layers. Can.

Reconstruction through optimization

,

인

로 나타낸다. 따라서, 상기 수학식 3에서 정의된 압축 화상 형성은 아래 <수학식 10>과 같이 다시 나타낼 수 있다.The present invention is a hyperspectral image

,

sign

It is represented by. Accordingly, the compressed image formation defined in Equation 3 may be expressed again as <Equation 10> below.

[Equation 10]

는 비선형 연산자이다.Here, Equation 10 is similar to a linear combination of an overcomplete dictionary in sparse coding, but the decoder in the present invention

Is a nonlinear operator.

에서

이므로, 상기 수학식 10은 불안정한 시스템을 의미한다. 이것은 관찰 결과로부터 하이퍼스펙트럴 이미지를 재구성하는 데에 문제를 야기할 수 있기 때문에 본 발명은 아래 <수학식 11>과 같이 목적 함수를 사용하여 하이퍼스펙트럴 재구성을 공식화할 수 있다.

in

Therefore, Equation 10 means an unstable system. Since this may cause problems in reconstructing the hyperspectral image from the observation results, the present invention can formulate the hyperspectral reconstruction using an objective function as shown in <Equation 11> below.

[Equation 11]

는 인코더를 의미하고,

는 공간 그래디언트 연산자를 의미하며,

과

는 데이터 충실도와 prior항 사이의 상대적 중요성을 조정 또는 평가할 수 있다.here,

Means an encoder,

Means the spatial gradient operator,

and

Can adjust or evaluate the relative importance between data fidelity and prior terms.

-충실도는 자동 인코더 표현을 본 발명의 최적화 문제와 연관시킬 수 있기 때문에 본 발명의 목적 함수에 중요한 기여를 한다. 이는 도 9에 도시된 바와 같이 재구성된 이미지의 스펙트럴 정확도에 큰 영향을 미칠 수 있다. 도 9는 본 발명의

-충실도 프라이어의 영향에 대한 일 예시도를 나타낸 것으로, 도 9a는

가 없는 복원 결과를 나타낸 것이고, (b)는 이전이 결과를 나타낸 것이다. 인셋(inset)은 실측값과 비교된 피크 신호대 잡음비(PSNR)와 구조적 유사성(SSIM)을 나타낸 것으로, 프라이어가 PSNR과 SSIM에서 모두 크게 증가한 것을 알 수 있다. 또한, 프라이어가 반복 회수에 따라 정확도가 어떻게 증가하는지 알 수 있으며, 노란색 깃털과 붉은 깃털에 대한 스펙트럴 정확도 또한 알 수 있다.The first of the fryer terms uses the encoder-decoder pair to establish the accuracy of the nonlinear representation, while the second of the fryer term is the L1-norm regularizer of the total variation and prefers sparsity of the spatial domain. Fryer protest first

-Fidelity makes an important contribution to the objective function of the present invention as it can correlate automatic encoder representation with the optimization problem of the present invention. This can greatly affect the spectral accuracy of the reconstructed image, as shown in FIG. 9. 9 is the present invention

-Fidelity shows an example of the effect of the fryer, Figure 9a

Shows the restoration result without, and (b) shows the previous result. The inset shows the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) compared to the measured values, and it can be seen that the fryer significantly increased in both PSNR and SSIM. In addition, it is possible to see how the fryer increases in accuracy with the number of repetitions, and also the spectral accuracy for yellow and red feathers.

optimization

Since the total variation gradient sparse term cannot be differentiated, the objective function of Equation 11 is divided into two problems, <Equation 12> and <Equation 13> below.

[Equation 12]

[Equation 13]

Therefore, the optimization problem of the present invention can be expressed as <Equation 14> below.

[Equation 14]

Here, z may mean a spatial gradient of the reconstructed hyperspectral image.

The present invention repeatedly solves the problem of Equation 14 by using an alternating direction multiplier (ADMM) method as shown in Algorithm 1 shown below.

항은 라인 3에서 업데이트되고 ADAM 최적화 알고리즘에 의해 최소화된다. 근사 그래디언트 디센트를 사용하여 라인 4의 보조 변수

로

항을 최소화하고, 라인 5에 나타난 엘리먼트-와이즈 소프트-스레스홀딩(element-wise soft-thresholding) 함수

를 사용하여 이 항을 업데이트한다. 알고리즘1에서 그래디언트

제약의 희소성의 강도를 아래 <수학식 15>와 같이 제어할 수 있다.first,

The term is updated in line 3 and minimized by the ADAM optimization algorithm. Secondary variable in line 4 using approximate gradient descent

in

Minimize terms, and the element-wise soft-thresholding function shown in line 5

Use to update this term. Gradient in Algorithm 1

The strength of the scarcity of the constraint can be controlled as shown in Equation 15 below.

[Equation 15]

는 상기 수학식 14의 제약 조건을 충족시키기 위해 그래디언트 상승을 통해 라인6에서 업데이트된다. 이 과정은 정지조건에 도달할 때까지 반복된다. 솔루션 표현

가 획득되면 디코더 D(

)로 사용하여 최종 하이퍼스펙트럴 이미지를 재구성한다.Lagrange multiplier

Is updated in line 6 through gradient elevation to satisfy the constraint of Equation (14). This process is repeated until a stop condition is reached. Solution representation

Decoder D(

) To reconstruct the final hyperspectral image.

parameter

을 0.1로 설정하고, 알고리즘 1은

및 ρ을

과

으로 각각 설정한다. 최적화 도구는 약 20 회의 ADMM 반복을 수행한다. 상기 수학식 14의

에 대한 ADAM 최적화 프로그램은

의 학습 속도로 200 단계를 반복한다.In Equation 11 above, for nonlinear expression fidelity

Is set to 0.1, algorithm 1 is

And ρ

and

Respectively. The optimization tool performs about 20 ADMM iterations. Equation 14 above

The ADAM optimizer for

Repeat steps 200 at a learning rate.

Time complexity

이며, 여기서

는 커널 크기이고

와

는 컨볼루션의 입력과 출력을 위한 특징 맵의 수이다.

커널을 가진 컨볼루션 자동 인코더(11개의 숨겨진 레이어가 있는 64개 특징)에서 총 곱셈의 수는 약

이다. 기존의 데이터 구동 방식과 비교하여, 희소 코딩 방법은 숨겨진 계층이나 활성화 기능이 없는 얕은 컨볼루션 신경망으로 간주 될 수 있다.

하이퍼스펙트럴 이미지 패치의 6200개의 원자로 된 사전을 사용하면, 곱셈의 추정된 수는 2배 더 큰

이다.The time complexity of hyperspectral image reconstruction is proportional to the number of multiplications performed by the convolutional automatic encoder. When performing one-stride convolution, the number of multiplications for the convolutional layer is

Where

Is the kernel size

Wow

Is the number of feature maps for convolution input and output.

The total number of multiplications in a convolutional automatic encoder with a kernel (64 features with 11 hidden layers) is approximately

to be. Compared to the existing data driving method, the sparse coding method can be regarded as a shallow convolutional neural network without a hidden layer or an activation function.

Using the 6200 atomic dictionary of the hyperspectral image patch, the estimated number of multiplications is twice as large.

to be.

Activation at the encoder

가 희소성으로 제한되지 않음을 나타낸다. 본 발명은 희소성을 명시적으로 부과하지는 않지만,

가 ReLU 활성화 함수로 다른 레이어를 통과하는 동안 자동 인코딩은 표현을 희소하게 만든다. 이 희소성 부재의 또 다른 이점은 비선형 최적화를 단순화한다는 것이다. 인코더의 출력 계층에 ReLU 활성화 함수를 추가하는 것은 상기 수학식 11에서 음이 아닌 제약 조건 만족항을 추가적으로 요구한다. 게다가 보조 변수와 라그랑지안 변수인 두 개의 변수가 상기 수학식 14의 ADMM 공식에 도입되어야 하므로 결과적으로 수렴 속도가 느려질 수 있다.4 and the above, the ReLU activation function does not exist in the output layer of the encoder. This is a non-linear representation of the hyperspectral image

Indicates that it is not limited to scarcity. The present invention does not explicitly impose scarcity,

Auto-encoding makes the representation sparse while it passes through different layers with the ReLU activation function. Another advantage of this sparse member is that it simplifies nonlinear optimization. Adding the ReLU activation function to the output layer of the encoder additionally requires a non-negative constraint satisfaction term in Equation (11). In addition, since two variables, an auxiliary variable and a Lagrangian variable, must be introduced into the ADMM formula of Equation 14, convergence may be slow as a result.

Global vs. Local optimization

Global optimization approaches such as TwIST and SpaRSA are more effective at reconstructing spectral information, but local optimization techniques such as sparse coding operate each patch independently, preserving the image structure well. However, the amount of variance is limited by the patch size, which greatly affects the computational cost. The approach of the present invention combines the advantages of local and global optimization through automatic convolutional encoding and total variation terms.

는 영상 형성 모델에 따라 달라질 수 있다.To evaluate the performance of the reconstruction algorithm of the present invention, the present invention first creates coded test sets from existing Harvard and Columbia spectral image data sets, and uses three main encoding types: SDCASSI, DD-CASSI, and SS-CASSI. The imaging process can be simulated to create a new data set as shown in FIG. 12. Here, the matrix of Equation (11)

May vary depending on the image forming model.

The present invention compares the results of this technique with the results for TwIST, SpaRSA, and sparse coding, which are three other methods representing three different encoding architectures. The present invention selects the best architecture for each method, DD-CASSI for Twist and SpaRSA, and SS-CASSI for sparse coding and the technique of the present invention.

The reconstruction technique of the present invention shows a significant improvement in both spectral and spatial accuracy and is the fastest of the three. 6 shows an exemplary diagram for an average result for a Columbia image data set. In addition, the present invention provides an analysis of the parameter space, directly compares the method of the present invention with actual values, compares it with a learning-based reconstruction, introduces a new hyperspectral data set, and results with a real hyperspectral imaging system. To present. Finally, the present invention proposes two methods in multi-hyperspectral interpolation and hyperspectral demosaicing without hardware modification.

Spectral Accuracy vs. spatial resolution

Existing reconstruction techniques share an essential trade-off between spectral accuracy and spatial resolution that defines the quality of the final image. As shown in Fig. 1, conventional optimization approaches such as TwIST and SpaRSA yield good results in spectral accuracy, but sacrifice the sub-optimal spatial resolution. On the other hand, data-based approaches based on sparse coding provide good spatial resolution, but at the expense of spectral accuracy. Conversely, the method of the present invention produces high quality results in both domains. Here, the performance of the reconstruction algorithm of the present invention is compared in terms of spectral accuracy and spatial resolution, respectively.

Spectral accuracy

The present invention evaluates the spectral accuracy of the reconstructed spectral image by calculating the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). FIG. 7 shows a contrast comparison of a color checker hyper spectral image of the Columbia data set, and the present invention converts the result of the spectral image to sRGB through a calibrated 2-degree CIE color matching function for visualization. Average PSNR and SSIM results (38.87dB / 0.98) of the present invention in 31 wavelength channels are better than all reconstruction results of TwIST (31.57dB / 0.94), SpaRSA (30.59dB / 0.94) and sparse coding (28.85db / 0.92) You can see that In addition, as a result of evaluating the reconstructed spectral reflectance of the five primary colors (blue, green, red, yellow, and pink) on the chart, it can be seen that the results of the present invention are consistently closer to the measured values than the other methods. Is the root-mean-squared errors (RMSE) and averages for each color patch.

Spatial resolution

The present invention evaluates the spatial resolution of the spectral image reconstructed by the present invention by calculating the spatial frequency response as a modulation transfer function (MTF). The standard spatial frequency measurement chart (ISO 12233) is reconstructed again using Twist, SpaRSA, sparse coding, and the method of the present invention. 8 shows the results of reconstructing a standard spatial frequency measurement chart (ISO 12233) using Twist, SpaRSA, sparse coding, and the method of the present invention. As with other existing optimization methods, TwIST and SpaRSA show sub-optimal reconstruction of spatial frequencies, and data-centric approaches based on sparse coding improve this spatial resolution, so the method of the present invention yields the best results. You can see that

Parameter analysis

The effect of fidelity fryer

정확도이며 트레이닝된 자동 인코더 장치의 비선형 표현과 재구성 문제를 연관 짓고 있다. 도 9에 도시된 바와 같이, 이 프라이어는 재구성 정확도에 큰 영향을 미치는 것을 알 수 있다. SSN은 0.93에서 0.96으로 증가하는 반면 PSNR은 30.84dB에서 34.33dB로 크게 증가한다. 또한, 두 번째 행에서 이전 패턴이 노란색 및 붉은 깃털에 대한 반사 정확도 뿐 아니라 반복 횟수에 따라 PSNR에 어떻게 영향을 미치는지 알 수 있다.One of the major novelty in the optimization formula of the present invention is the equation (11)

It is an accuracy and correlates the reconstruction problem with the nonlinear representation of the trained automatic encoder device. As shown in Fig. 9, it can be seen that this fryer greatly influences the reconstruction accuracy. The SSN increases from 0.93 to 0.96, while the PSNR increases significantly from 30.84dB to 34.33dB. Also, in the second row, you can see how the previous pattern affects PSNR according to the number of iterations as well as the reflection accuracy for yellow and red feathers.

hidden Layered effect

FIG. 10 shows the effect of the number of hidden layers on the spatial resolution of reconstruction. As shown in FIG. 10A, it can be seen that the accuracy of reconstruction is not significantly improved in 11 or more layers. Since the spatial resolution does not increase significantly, the number of hidden layers can be set to 11 after considering the trade-off between performance and memory.

The present invention undergoes additional experiments by beta process element analysis (BPFA) and sub-grade reconstruction, and the quantitative evaluation of the present invention is calculated for reflectivity rather than luminosity to ensure chromaticity. This generally lowers the PSNR value by 2.0 to 3.0 dB compared to the luminance. In BPFA, hyperspectral images are reconstructed from coded inputs by employing the reconstruction method used for blind compression detection. The latter method improves the initial estimation of the hyperspectral image using the spectral-spatial correlation present in similar and non-localized hyperspectral image patches. As shown in FIG. 13, the average PSNR and SSIM measurements of 32 hyperspectral images of the Columbia data set are 21.71 dB and 0.69 for BPFA, 24.48 dB and 0.85 for low rank reconstruction, and in the method of the present invention, It can be seen that 32.46 dB and 0.95 were obtained, respectively. BPFA was designed for multi-frame CASSI, but only a single input was used for fairness of comparison. As described above, the quality of the lower grade reconstruction depends on the initial estimation, in this case, the TwIST reconstruction. Therefore, the PSNR and SSIM values of the low rank reconstruction may be higher than or equal to TwIST (23.74 dB / 0.85) shown in FIG. 6.

는 회귀 모델에서 암묵적으로 부호화될 수 있다. 본 발명은 재구성을 회귀 기반 네트워크와 비교한다. 원래 하이퍼스펙트럴 이미징을 위해 설계되지 않았기 때문에 압축 입력에서 하이퍼스펙트럴 이미지를 직접 추정하는 심층 컨볼루션 회귀 네트워크를 학습하도록 수정하고, 수정된 심화 회귀 네트워크는 64개의 기능맵으로 구성된 11개의 숨겨진 컨볼루션 레이어로 구성될 수 있다. 컨볼루션은 ReLU 활성화 기능을 사용하여

커널로 수행될 수 있으며, 학습을 위해 콜럼비아 데이터 세트를 사용할 수 있으며 이미지의 크기는

일 수 있다. 회귀 네트워크는 네트워크에서 고정 변조 행렬

를 암묵적으로 인코딩하기 때문에

이미지로 제한된다. 본 발명은 ADAM 최적화 도구를 사용하여 네트워크를 학습할 수 있으며, 배치로 8개의 이미지를 패킹하고 3000 에포크 동안 트레이닝을 수행할 수 있다.Direct learning-based reconstruction trains an end-to-end regression network that outputs the original image using compression measurements as input. Modulation matrix

Can be implicitly coded in the regression model. The present invention compares reconstruction to a regression based network. Since it was not originally designed for hyperspectral imaging, it is modified to train a deep convolutional regression network that directly estimates the hyperspectral image from the compressed input, and the modified deep regression network consists of 11 hidden convolutions consisting of 64 functional maps. It can be composed of layers. Convolution uses the ReLU activation feature

Can be done with the kernel, you can use the Columbia data set for training and the size of the image

Can be The regression network is a fixed modulation matrix in the network.

Because it implicitly encodes

Limited to images. The present invention can train the network using the ADAM optimization tool, pack 8 images in batches and perform training for 3000 epochs.

It uses 32 spectral images from the Columbia data set for comparison, and it is very fast (average 0.14 seconds) because regression-based reconstruction does not require an optimization step, but it is significantly less accurate in both spectral and spatial domains. Yield a reconstruction. 13 shows representative results for this, and FIG. 14 shows PSNR and SSIM values averaged over the entire data set. As can be seen from FIGS. 13 and 14, end-to-end regression in addition to low quality of regression-based reconstruction Learning is very impractical in that you have to train a different model whenever the image settings (image size, mask pattern, pixel pitch of the lens or sensor) change.

new Hyperspectral Image data set

It can be seen that the Columbia data set provides images with a wide range of spectral information, while providing low spatial resolution and slightly out of focus images, likewise Harvard data sets provide high spatial resolution but spectral range It can be seen that it is limited. To improve this, you can capture a new high-resolution data set consisting of 30 hyperspectral images covering a wide spectral range, and Figures 11 and 12 show some examples, and the complete data set can be downloaded from the project website. can do.

real Hyper Spectral Camera results

및 파장 의존 픽셀 시프트 함수

와 같은 시스템의 광학 특성을 보정할 수 있다.In order to verify the validity of the reconstruction algorithm, a prototype of the spatial spectral encoded DD-CASSI imaging system may be created as shown in FIG. 15A. Here, the system can consist of an apochromatic lens, a relay lens, two prisms (NBK-7, 2 degree angle, 13 pixel dispersion generation), a coded aperture and a CCD imaging sensor. All relay lenses (Sigma A, f/1.4) have the same focal length (50mm) for one-to-one imaging. The camera may be a point graygrass hopper (GS3 9.1MP Mono) with a pixel pitch of 3.69 μm. The coded aperture mask includes any binary pattern made by lithographic chrome etching on a quartz plate with a pixel pitch of 7.40 μm in the binary pattern. The pixels of the mask correspond to 2 x 2 pixels in the imager's CCD sensor. The screen is captured by a solid state plasma light source. Binary mask pattern in Equation 2

And wavelength dependent pixel shift function

The optical characteristics of the system can be corrected.

Learning lighting constancy

(CIE A, 텅스텐),

(형광등),

(CIED 50),

(CIE D65) 및

(플라스마)의 5가지 다른 색온도에서 컬럼비아의 데이터 세트의 32가지 하이퍼스펙트럴 반사 이미지를 사용하여 192개의 하이퍼스펙트럴 이미지에 대한 추가 학습 데이터 세트를 생성할 수 있다. 이 데이터 세트는 스케일 불변성을 위해 추가로 추가되어 총 384개의 새로운 하이퍼스펙트럴 이미지가 생성할 수 있다. 본 발명은 이 증가한 데이터 세트로부터 96 x 96 x 31사이즈의 19,200개의 텐서 패치를 샘플링할 수 있다.The model is retrained with additional datasets under various lights at different color temperatures to handle the actual inputs in the prototype. The present invention

(CIE A, tungsten),

(Fluorescent lamp),

(CIED 50),

(CIE D65) and

You can create additional training datasets for 192 hyperspectral images using 32 hyperspectral reflection images from Columbia's dataset at five different color temperatures in (Plasma). This data set can be further added for scale invariance, resulting in a total of 384 new hyperspectral images. The present invention can sample 19,200 tensor patches of size 96 x 96 x 31 from this increased data set.

Results for actual data

FIG. 15B compares the reconstruction using the method of the present invention with TwIST4, and the plot of FIG. 15C compares the spectral accuracy of the selected patch reconstructed by each method and the actual value measured by a spectrophotometer. 15, it can be seen that the spectral reconstruction of the present invention surpasses the conventional approach. TwIST reconstruction has difficulty in reconstructing artifacts in space and can be seen to be less accurate in the spectral region, and FIGS. 15D and 15E show the reconstruction results of the present invention.

Hyperspectral Interpolation

를 파장 서브 샘플링 행렬로 단순히 대체한다.The method of the present invention allows interpolation of a multi-spectral image into a high-resolution hyperspectral image without hardware modification using a spectral fryer. Measurement matrix of Equation (11)

Is simply replaced by the wavelength subsampling matrix.

FIG. 16A shows the interpolation result of sub-sampling 52%, 26%, and 10% of the original spectral wavelength, and is converted into 16, 8, and 3 channels, respectively. The interpolated reconstruction for 31 wavelengths is compared with the actual value (or actual value), and the accuracy of reconstruction is high, but when only 10% of the information is used, the accuracy of reconstruction decreases.

Hyperspectral Demosaicing

행렬을 대체한다. 본 발명은

행렬에 대한 가우시안 블러(Gaussian blur)로서 회절 흐림을 설명하고 재구성 흐림을 피하기 위해

를 매우 작은 값

으로 설정한다. 도 16b는 본 발명의 스펙트럴 재구성이 얼마나 정확한지 나타낸 것으로, 본 발명은 디지털 카메라의 디모자이싱(demosaicing)과 유사하게 바이엘 패턴의 다중 스펙트럴 입력을 사용하여 단일샷 하이퍼스펙트럴 이미징을 가능하게 한다.The present invention extends the interpolation reconstruction to enable hyperspectral demosaicing assuming that the inputs correspond only to the wavelengths of 450nm, 520nm, 580nm and 650nm according to the existing buyer pattern, and the spatial and spectral of Equation 11 With subsampling matrix

Replace the matrix. The present invention

Gaussian blur for the matrix to explain the diffraction blur and avoid reconstruction blur

Very small value

Set to 16B shows how accurate the spectral reconstruction of the present invention is, and the present invention enables single-shot hyperspectral imaging using multiple spectral inputs of a Bayer pattern, similar to demosaicing of digital cameras. .

limit

The reconstruction algorithm of the present invention includes a total variation term (associating spectral information with neighboring pixels) that favors scarcity in the spatial domain. If the input quality is not sufficient, a very fine image structure can be reconstructed as the next best thing. It can be seen that the input image is out of focus as shown in FIG. 17. Although the method of the present invention shows better results than other approaches, it can be seen that it is not perfect for reconstructing the small details of the printed word.

As such, it can be seen that the technique according to the present invention surpasses existing methods for spatial resolution and spectral accuracy. Specifically, the present invention is to train a natural spectral in a nonlinear expression using a convolutional automatic encoder, and to formulate a new nonlinear optimization using the result of the automatic encoder as a spectral fryer. The reconstruction method of the present invention can be applied to all compression imaging architectures, and also the computational complexity can be reduced by a factor of two compared to the best method based on sparse coding.

As described above, the technology according to the present invention receives coded data for an image, for example, an encoded sensor input, and images for coded data based on a pre-generated learning model, for example, a nonlinear learning model. By reconstructing the hyperspectral image of, high-quality hyperspectral image can be reconstructed and applied to any existing compression imaging architecture.

In addition, the technique according to the present invention may reconstruct a hyperspectral image of an image for coded data based on an optimization technique, for example, a nonlinear optimization technique, as well as a pre-generated learning model. That is, the present invention uses artificial intelligence, for example, a type of convolutional neural network (CNN) and a nonlinear learning model, a convolutional autoencoder, and a nonlinear optimization technique jointly combined and used. , Through this, it is possible to reconstruct a high quality hyperspectral image.

The method according to the present invention may be configured as an apparatus or system, and the configured apparatus or system may include all functions of the method according to the present invention described above.

The system or device described above may be implemented with hardware components, software components, and/or combinations of hardware components and software components. For example, the systems, devices, and components described in the embodiments include, for example, processors, controllers, arithmetic logic units (ALUs), digital signal processors (micro signal processors), microcomputers, and field programmable arrays (FPAs). ), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose computers or special purpose computers. The processing device may run an operating system (OS) and one or more software applications running on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For convenience of understanding, a processing device may be described as one being used, but a person having ordinary skill in the art, the processing device may include a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that may include. For example, the processing device may include a plurality of processors or a processor and a controller. In addition, other processing configurations, such as parallel processors, are possible.

The software may include a computer program, code, instruction, or a combination of one or more of these, and configure the processing device to operate as desired, or process independently or collectively You can command the device. Software and/or data may be interpreted by a processing device, or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. , Or may be permanently or temporarily embodied in the transmitted signal wave. The software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer readable recording media.

The method according to the embodiments may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, or the like alone or in combination. The program instructions recorded in the medium may be specially designed and configured for an embodiment or may be known and usable by those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs, DVDs, and magnetic media such as floptical disks. -Hardware devices specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language code that can be executed by a computer using an interpreter, etc., as well as machine language codes produced by a compiler. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described by a limited embodiment and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques are performed in a different order than the described method, and/or the components of the described system, structure, device, circuit, etc. are combined or combined in a different form from the described method, or other components Alternatively, even if replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (23)

delete Receiving coded data for an image; And
Reconstructing a hyperspectral image of an image for the coded data based on a pre-generated learning model
It includes,
Generating a nonlinear learning model through learning using a preset hyperspectral image data set
Further comprising,
Reconstructing the hyperspectral image
Reconstructing the hyperspectral image based on the generated nonlinear learning model and nonlinear optimization technique,
Reconstructing the hyperspectral image
Hyperspectralization of the image for the coded data based on the nonlinear optimization technique that jointly normalizes the accuracy of the nonlinear spectral representation for the generated nonlinear learning model and the sparsity of gradients in the spatial domain. A hyperspectral image reconstruction method comprising reconstructing an image.
According to claim 2,
The step of generating the nonlinear learning model is
A hyperspectral image reconstruction method characterized by generating a nonlinear learning model by learning a convolutional automatic encoder through training using the hyperspectral image data set.
According to claim 3,
The step of generating the nonlinear learning model is
A hyperspectral image reconstruction method comprising learning the encoder network in a nonlinear space using the convolution automatic encoder and then generating the nonlinear learning model using a decoder network.
According to claim 4,
The convolution automatic encoder
And an encoder network for converting the hyperspectral image data set into a nonlinear representation and a decoder network for generating an original data set from the nonlinear representation.
According to claim 2,
The non-linear learning model
A hyperspectral image reconstruction method, characterized in that it is a learning model that outputs a nonlinear reconstruction of the hyperspectral image.
delete Receiving coded data for an image; And
Reconstructing a hyperspectral image of an image for the coded data based on a pre-generated learning model
It includes,
Generating a nonlinear learning model through learning using a preset hyperspectral image data set
Further comprising,
Reconstructing the hyperspectral image
Reconstructing the hyperspectral image based on the generated nonlinear learning model and nonlinear optimization technique,
Reconstructing the hyperspectral image
A hyperspectral image reconstruction method comprising reconstructing a hyperspectral image of an image for the coded data by repeatedly performing the nonlinear optimization technique using an alternating direction multiplier (ADMM) method.
According to claim 2,
The receiving step
A method of reconstructing hyperspectral images, characterized by receiving coded data for the images using a compressed hyperspectral imaging approach.
Receiving coded data for an image; And
Reconstructing a hyperspectral image of an image for the coded data based on a previously learned spectral fryer
It includes,
Learning the spectral prior using a preset hyperspectral image data set
Further comprising,
Reconstructing the hyperspectral image
Reconstructing the hyperspectral image based on the learned spectral fryer and nonlinear optimization technique,
Reconstructing the hyperspectral image
Hyperspectralization of the image for the coded data based on the nonlinear optimization technique that jointly normalizes the accuracy of the nonlinear spectral representation for the learned spectral fryer and the sparsity of gradients in the spatial domain. Hyperspectral image reconstruction method for reconstructing images.
delete delete Receiving coded data for an image; And
Reconstructing a hyperspectral image of an image for the coded data based on a previously learned spectral fryer
It includes,
Learning the spectral prior using a preset hyperspectral image data set
Further comprising,
Reconstructing the hyperspectral image
Reconstructing the hyperspectral image based on the learned spectral fryer and nonlinear optimization technique,
Reconstructing the hyperspectral image
A hyperspectral image reconstruction method comprising reconstructing a hyperspectral image of an image for the coded data by repeatedly performing the nonlinear optimization technique using an alternating direction multiplier (ADMM) method.
delete A receiver configured to receive coded data for an image; And
A reconstruction unit reconstructing a hyperspectral image of an image of the coded data based on a previously generated learning model
It includes,
A generation unit that generates a nonlinear learning model through learning using a preset hyperspectral image data set
Further comprising,
The reconstruction unit
Reconstructing the hyperspectral image based on the generated nonlinear learning model and nonlinear optimization technique,
The reconstruction unit
Hyperspectralization of the image for the coded data based on the nonlinear optimization technique that jointly normalizes the accuracy of the nonlinear spectral representation for the generated nonlinear learning model and the sparsity of gradients in the spatial domain. Ultra-spectral image reconstruction apparatus, characterized in that to reconstruct the image.
The method of claim 15,
The generation unit
An apparatus for reconstructing hyperspectral images, wherein the nonlinear learning model is generated by learning a convolutional automatic encoder through learning using the hyperspectral image data set.
The method of claim 16,
The generation unit
An apparatus for reconstructing hyperspectral images, wherein the encoder network is trained in a nonlinear space using the convolution automatic encoder, and then the decoder network is generated using a decoder network.
The method of claim 17,
The convolution automatic encoder
And an encoder network for converting the hyperspectral image data set into a nonlinear representation and a decoder network for generating an original data set from the nonlinear representation.
The method of claim 15,
The non-linear learning model
A hyperspectral image reconstruction apparatus, characterized in that it is a learning model that outputs a nonlinear reconstruction of the hyperspectral image.
delete A receiver configured to receive coded data for an image; And
A reconstruction unit reconstructing a hyperspectral image of an image of the coded data based on a previously generated learning model
It includes,
A generation unit that generates a nonlinear learning model through learning using a preset hyperspectral image data set
Further comprising,
The reconstruction unit
Reconstructing the hyperspectral image based on the generated nonlinear learning model and nonlinear optimization technique,
The reconstruction unit
A hyperspectral image reconstruction apparatus characterized by reconstructing a hyperspectral image of an image for the coded data by repeatedly performing the nonlinear optimization technique using an alternating direction multiplier (ADMM) method.
The method of claim 15,
The receiving unit
An apparatus for reconstructing hyperspectral images, characterized by receiving coded data for the images using a compressed hyperspectral imaging approach.
delete

Images