KR20180137795A - Hyperspectral Imaging Device - Google Patents

Abstract

Disclosed is an imaging device. According to one embodiment of the present invention, the imaging device comprises: a multi-aperture which modulates an input source in a spatial domain; a random filter array for modulating the input source modulated in the multi-aperture in a spectrum domain; and a photo-sensor which integrates the modulated source in accordance with a spectrum range sensitivity.

Description

Hyperspectral Imaging Device < RTI ID = 0.0 >

The present invention relates to a hyperspectral imaging device. To a hyper-spectral imaging device based on digital signal processing.

Spectral imaging is a technique that integrates existing imaging and spectral methods to obtain spatial and spectral information of an object. Spectral imaging was defined by Goetz at the end of the 1980s and was discussed to remotely sense the earth. Spectral imaging can be divided into multispectral imaging, hyperspectral imaging, and ultraspectral imaging, depending on spectral resolution, number of bands, width, and band continuity.

Multispectral imaging systems typically collect data in a relatively non-contiguous broad spectral range measured in micrometers or tens of micrometers. Although hyperspectral imaging can capture hundreds of spectral bands, ultrasound imaging systems collect more spectrum bands than hyperspectral imaging.

Figure 1 shows the concept of spectral imaging.

Figure 2 illustrates the functionality of imaging systems.

As shown in Figure 1, the spectral imaging data may be visualized as a three dimensional cube or a stack of two dimensional images due to its internal structure. Depth is a function of wavelength. Spectral imaging can obtain reflectance, absorption, or fluorescence spectrum for each pixel of an image.

Spectrum imaging technology is a technology applied in the field of remote sensing such as aircraft monitoring or satellite imaging and is also a technology applied in mining and geology, agriculture, military, environmental, global change research, forensics and medical.

Prior Art [1] Li, Qingli, et al. "Review of spectral imaging technology in biomedical engineering: achievements and challenges." Journal of biomedical optics 18.10 (2013): 100901-100901.

[2] F. H. Goetz et al., "Imaging spectrometry for Earth remote sensing," Science 228 (4704), 1147-1153 (1985).

[3] A. F. H. Goetz, "Three decades of hyperspectral remote sensing of the Earth: a personal view," Remote Sens. Environ. 113, S5 ?? S16 (2009).

Prior art literature [4] E. J. Cand ㅸ s, "Compressive sampling," Proc. Int. Congr. Mathematicians 3, 1433-1452 (2006).

Prior art [5] Hagen, Nathan, and Michael W. Kudenov. "Review of snapshot spectral imaging technologies." Optical Engineering 52.9 (2013): 090901-090901.

Prior Art [6] Cao, Xun, et al. "Computational Snapshot Multispectral Cameras: Toward dynamic capture of the spectral world." IEEE Signal Processing Magazine 33.5 (2016): 95-108.

Prior Art Document [7] A. Wagadarikar, R. John, R. Willett, and D. J. Brady, "Single dispersive design for coded aperture snapshot spectral imaging," Appl. Opt., Vol. 47, no. 10, pp. B44? B51, 2008.

[8] M. E. Gehm, R. John, D. J. Brady, R. M. Willett, and T. J. Schulz, "Singleshot compressive spectral imaging with a dual-disperser architecture, Opt. Express, vol. 15, no. 21, pp. 14013? 14027, 2007.

The spectral imaging device according to an embodiment of the present invention is more suitable for physical implementation than a general spectral imaging device and can be miniaturized.

An imaging apparatus according to an embodiment of the present invention includes a multi-aperture for modulating an input source in a spatial domain, a random filter array for modulating an input source modulated in the multi-aperture in the spectral domain, and a modulator for modulating the modulated source in a spectral range And includes a photosensor that integrates according to sensitivity.

The spectral imaging device according to an embodiment of the present invention is more suitable for physical implementation than a general spectral imaging device and can be miniaturized.

Figure 1 shows the concept of spectral imaging.
Figure 2 illustrates the functionality of imaging systems.
Figure 3 shows a method commonly used in spectral imaging.
FIG. 4 conceptually illustrates a process of spectral imaging performed by a spectral imaging apparatus according to an exemplary embodiment of the present invention.
5 shows a spectral imaging apparatus according to an embodiment of the present invention.
6 is a diagram comparing a multi-aperture and a general multi-aperture according to an embodiment of the present invention.
7 is a flow chart illustrating a spectral imaging method in accordance with an embodiment of the present invention.

Hereinafter, specific embodiments of the present invention will be described in detail with reference to the drawings. It should be understood, however, that there is no intention to limit the spirit of the present invention to the specific embodiments set forth below, and that those skilled in the art, having the benefit of the teachings of the present invention, Or the like, but it will also be included in the spirit of the present invention.

Figure 3 shows a method commonly used in spectral imaging.

As shown in FIG. 3, spectral imaging can be performed in the manner of whiskbroom, pushbroom, staring, and snapshot.

First, the whiskbroom method is briefly described.

The Whiskbroom imaging mode is also referred to as the point scanning method. A single point is scanned along two spatial dimensions (x and y) by moving the sample or detector. The reflected light is then dispersed by a prism and recorded by a linear array detector. A spectral image data cube is acquired by scanning a two-dimensional scene and dispersing it in the wavelength domain. The whiskbroom mode generally requires a two-axis motor position table to complete the image acquisition, which complicates the hardware configuration.

Second, we will briefly explain the pushbroom approach.

The pushbroom scheme is also referred to as line scanning. Unlike the whiskbroom method, which scans one point at a time, the pushbroom method has the advantage of collecting not only spatial information corresponding to each spatial point of a slit during scanning but also spectral information corresponding to each space simultaneously.

Third, we explain the staring method.

The staring scheme is also referred to as a band sequential scheme. The staring method is a spectral scanning method that acquires a single-band two-dimensional gray scale image including the entire spatial information at a time. In the staring mode, the entire spatial information passes through one spectral band at a time and is imaged on the focal plane, and images are stacked according to wavelength to form 3D data. This is different from whiskbroom and pushbroom.

Fourth, we explain the snapshot method.

The snapshot method is also referred to as a single-shot method. Snapshot is a method of recording space and spectral information in a single exposure to a zone detector. Snapshot is scanned in spatial or spectral dimension. Thus, unlike the whiskbroom, pushbroom, and staring modes, which have a limited periodic resolution, snapshot mode is an imaging method that does not require scanning at all, because it limits the time resolution. Snapshot mode can acquire a complete spectral data cube by simultaneously imaging the remapped and distributed image zones to the CCD detector.

Table 1 compares / explains the features of each method.

Hereinafter, computational spectral imaging will be described.

In recent years, spectral imaging is performed using a compression sensing theory. Compression sensing uses L1-standard reconstruction techniques to reconstruct poorly sampled data due to Nyquist constraints in general. For further details on compression sensing, see Prior Art Documents 4 and 5.

First, a brief description of a single dispersive coded aperture snapshot imager is given.

Referring to Prior Art 7, there is proposed an aperture snapshot imaging system coded using a single diffuser referred to as SD-SASSI. The spectral data cubes are modulated by the coded mask and variance. The light beams of different wavelengths are differently offset by the dispersive element after being modulated by the diaphragm code.

Secondly, a brief description of a dual dispersive coded aperture snapshot imager is provided.

Since a typical CASSI system forms a sheared three-dimensional spectral data cube, the observed snapshot is blurred by dispersion. To overcome this blurring effect, Gehm proposes a dual distributor structure that generates unsheared spectral codes with replicated slanted codes symmetrically placed on both sides of the coded aperture. Refer to Prior Art Document 8 for a detailed description related thereto.

Third, a spatial-spectral coded compressive spectral imager is briefly described.

The feasible codes for generic CASSI and non-spatial modulated imaging systems are limited by their physical modulation capabilities. Theoretically, 3-D-CASSI, which randomly encodes a spatial-spectral data cube, can achieve more feasible codes and higher performance than a typical CASSI.

Hereinafter, a spectral imaging apparatus according to an embodiment of the present invention will be described.

Recently, the requirements and applications for spectral imaging have increased in many fields. However, generic spectral imaging devices require much time to create a three-dimensional data cube. Also, the price of the spectral imaging device is expensive.

To overcome the above disadvantages, the proposed system CASSI or CASSI has a limitation in minimizing the size of the apparatus. In addition, there is a problem that physical implementation of CASSI is difficult. Therefore, one embodiment of the present invention for solving the problem will be described below. The spectral imaging device according to an embodiment of the present invention is suitable for physical implementation and can downsize the spectral imaging device.

FIG. 4 conceptually illustrates a process of spectral imaging performed by a spectral imaging apparatus according to an exemplary embodiment of the present invention. 5 shows a spectral imaging apparatus according to an embodiment of the present invention.

4 and 5, a spectral imaging device 1 according to an embodiment of the present invention includes a multi-aperture 10, a random filter array 20, and a CMOS sensor 30.

In general, the multi-aperture method refers to a method in which each aperture looks at a different portion with respect to one object. For example, suppose an object with four pixels looks at the multi-aperture. At this time, the first aperture looks at the first pixel, the second aperture shows the second pixel, the third aperture shows the third pixel, and the fourth aperture shows only the fourth pixel. Then, the images obtained through the respective apertures are gathered to constitute the whole image.

Meanwhile, one aperture according to an embodiment of the present invention acquires an image of a plurality of pixels to compress or modulate spatial information. For example, in the multi-aperture of the present invention, the first aperture may be regarded as 1 to 2 pixels, the second aperture may be regarded as 2 to 4 pixels, and the third aperture may be regarded as 3 to 4 pixels. . Here, the meaning of the description means that the aperture acquires the input source for the pixel of interest.

In yet another embodiment, a multi-aperture according to an embodiment of the present invention may use a general multi-aperture approach. The operation of the above-described general multi-aperture and the multi-aperture according to an embodiment of the present invention is shown in Fig.

6 is a diagram comparing a multi-aperture and a general multi-aperture according to an embodiment of the present invention.

(a) shows the operation of the general multi-aperture, and (b) shows the operation of the multi-aperture according to an embodiment of the present invention. (b), the lenses constituting the multi-aperture acquire the input source for the pixels of the object object redundantly, rather than all the lenses acquiring the input sources for the different pixels.

Returning to Figs. 4 to 5 again.

The multi-aperture (10) receives the input signal and modulates / compresses it. Specifically, the multi-aperture processor 10 compresses or modulates spatial information of a scene in a single snapshot. The spectral imaging apparatus 1 according to an embodiment of the present invention includes the multi-aperture 10 not including a single aperture so that spatial information of a scene for a plurality of points in a single scan can be compressed / can do. Referring to FIG. 5, a microlens is shown, and a set of microlenses can be referred to as a multi-aperture.

The random filter array 20 modulates the spectral information of the input signal compressed / modulated in the multi-aperture 10. Here, in one embodiment, the random filter array 20 may comprise a thin film optical filter comprised of multiple layers of high-index material and low-index material deposited on a substrate. Referring to FIG. 5, a spectral filter unit is shown, and a set of spectral filter units can be referred to as a random filter array.

The CMOS sensor 30 senses the modulated signal in the random filter array 20. In other words, the input signal modulated in the random filter array 20 is projected to the CMOS sensor 30. [ Referring to FIG. 5, a photo sensor unit is shown, and a set of photo sensor units may be referred to as a CMOS sensor.

The input signal passed through each random filter varies abruptly with respect to wavelength. Accordingly, the spectral imaging apparatus according to an exemplary embodiment of the present invention can be compactly designed without using a modulator, thereby facilitating modulation compared with a general spectral imaging apparatus. 7 is a flow chart illustrating a spectral imaging method in accordance with an embodiment of the present invention.

In a spectral imaging method according to an embodiment of the present invention, an input source is first modulated in a spatial domain through a multi-aperture (S101).

In spatial modulation, the relationship between a single aperture and the input signal X is expressed as: < EMI ID = 1.0 >

Where d _i is a spatially modulated output, a _i is a single aperture, and X is a hyperspectral image input.

And the modulated input source in the spatial domain is modulated in the spectral domain through the random filter array (S103).

In spectral modulation, the output d _i modulated through the random filter array 20 is expressed as: < EMI ID = 2.0 >

Here, y _i is the output sample of the random filter array in the i-th lens. T is the transmission set of the random filter. The i-th lens means a lens constituting the i-th aperture in the multi-aperture.

과 같이 행렬 Y가 표기될 때, 압축 스펙트럴 이미징 장치는 수학식 3과 같이 모델링 된다.

When the matrix Y is marked as shown in Equation (3), the compression spectral imaging apparatus is modeled as shown in Equation (3).

는 입력 소스이고, A는 멀티 어퍼쳐 매트릭스이고, T는 필터 어레이의 전송 매트릭스이다. X는 입력소스를 나타내는 N ㅧ P매트릭스이다. N은 파장의 범위이고, P는 픽셀의 개수이다. A는 공간 변조 현상을 설명하는 L ㅧ P매트릭스 이다. L은 어퍼쳐의 개수이고, P는 픽셀의 개수이다. i번째 어퍼쳐에 대하여 j번째 픽셀이 보이지 않으며, 행렬 A내 컴포넌트 a_ij가 0이다. j번째 픽셀이 i 번째 어퍼쳐에 대해 모두 보이면 행렬 A내 컴포넌트 a_ij가 1이다. 그렇지 않으면 a_ij는 0과 1사이에 있다. T는 M ㅧ N의 랜덤 필터 어레이의 전송 매트릭스이다. M은 필터의 개수이고, N은 파장의 범위이다. 전송 매트릭스 T는 수학식 4와 같이 표현된다.From here

A is the multi-aperture matrix, and T is the transmission matrix of the filter array. X is an N ㅧ P matrix representing the input source. N is a wavelength range, and P is the number of pixels. A is an L ㅧ P matrix describing spatial modulation phenomena. L is the number of apertures, and P is the number of pixels. The jth pixel is not visible for the ith aperture, and the component a _ij in matrix A is zero. The j-th pixels visible for both the i-th aperture A is a matrix component within a _ij 1. Otherwise a _ij is between 0 and 1. T is the transmission matrix of the random filter array of M N. M is the number of filters, and N is the wavelength range. The transmission matrix T is expressed by Equation (4).

In conclusion, the dimension of Y is M ㅧ L.

The modulated sources are integrated along with the spectral range sensitivity of the CMOS sensor 30 (S105). Specifically, the light source that passes through the multi-aperture and passes through the random filter is integrated with the wavelength efficiency of the CMOS sensor.

The sensed modulated signal is reconstructed in a signal reconstruction unit (not shown) (S107).

Hereinafter, a process of reconstructing a modulated source according to an embodiment of the present invention will be described.

Compression sensing theory is used to solve the inverse problem when the sensing matrix is not confirmed. This is expressed by Equations (5) and (6).

는 N ㅧ L매트릭스이다.From here

Is the N ㅧ L matrix.

In the matrix T, the row size M is smaller than the column size N. [ Expression (6) in the column direction is expressed by Equation (7).

Where d is expressed as Equation 8 using basis Ψ and a sparse vector.

Using Equation (8), Equation (7) is expressed again as Equation (9).

The compression sense reconstruction algorithm can be used to find a sparse approximation of the spatial information of an aperture. This is expressed as Equation 10 as an optimization problem.

를 추정할 수 있다. 그리고, D를

라고 하면, 수학식 5는 수학식 11로 다시 표현된다.And,

Can be estimated. And, D

, Equation (5) is again expressed by equation (11).

Transposing both sides of Equation (11) is expressed as Equation (12).

For a matrix A, the row size L is smaller than the column size P. Therefore, equation (12) can be expressed as shown in equation (13) in the column direction.

And applying the optimization problem here is expressed as the following equation (14).

를 트랜스포즈하면 변조전 최초 입력 장면인

를 재건할 수 있다.Then,

The first input scene before modulation

Can be reconstructed.

Up to now, to facilitate understanding of the present invention, an exemplary embodiment of a spectral imaging apparatus has been described and shown in the accompanying drawings. It should be understood, however, that such embodiments are for the purpose of illustrating the invention only, and not for limitation. And it is to be understood that the invention is not limited to the details shown and described. Since various variations can occur to those of ordinary skill in the art.

Claims (11)

A multi-applicator for modulating the input source in the spatial domain;
A random filter array for modulating the input source modulated in the multi-aperture in the spectral domain; And
And a photosensor that integrates the modulated source according to spectral range sensitivity
Spectral imaging device. The method according to claim 1,
The multi-aperture unit compresses / modulates spatial information for a plurality of parts in one scan with respect to the object
Spectral imaging device. 3. The method of claim 2,
Wherein the multi-aperture lens comprises a first lens and a second lens located on the same plane,
An input source for a pixel of an object to be acquired by the first lens and an input source for a pixel of an object to be acquired by the second lens are partially overlapped
Spectral imaging device. The method according to claim 1,
Wherein the random filter array comprises a plurality of layers,
Wherein the plurality of layers comprises a first layer comprising a material having a relatively high refractive index and a second layer comprising a material having a relatively low refractive index
Spectral imaging device. The method according to claim 1,
Wherein the photosensor is configured such that the input source through the multi-aperture and the local filter array integrates with the wavelength efficiency of the photosensor
Spectral imaging device. The method according to claim 1,
And a signal reconstruction unit reconstructing the modulated signal integrated in the photo sensor
Spectral imaging device. Modulating an input source in a spatial domain;
Modulating the modulated input source in the spatial domain in the spectral domain; And
And integrating the modulated source in the spectral domain according to spectral range sensitivity
A method of operating a spectral imaging device. 8. The method of claim 7,
Wherein modulating the input source in the spatial domain comprises:
Comprising compressing / modulating spatial information for a plurality of parts in a single scan with respect to an object of interest
A method of operating a spectral imaging device. 8. The method of claim 7,
The step of integrating the modulated source in the spectral domain according to spectral range sensitivity comprises:
Wherein the input source passed through the multi-aperture and the local filter array includes integrating with the wavelength efficiency of the photosensor
A method of operating a spectral imaging device. The method according to claim 1,
Further comprising rebuilding the integrated modulated signal
A method of operating a spectral imaging device. A program for implementing the algorithm of any one of claims 7 to 10.

Images