JP6923897B2 - Filters, imaging devices and imaging systems - Google Patents

Description

The present invention relates to filters for optical instruments, imaging devices and imaging systems.

In recent years, a multispectral image (MSI) having more spectral information than a color image, that is, an RGB image has attracted attention. MSI enables faithful color reproduction without illumination and object recognition at a spectral level that cannot be visually recognized by humans. Therefore, MSI is expected to be applied in various fields such as remote sensing, pathological diagnosis, and sugar content measurement of fruits and vegetables. For example, attempts have been made to improve the classification accuracy by performing tissue classification in MSI of liver pathological specimens.

Various MSI imaging systems have been proposed. For example, an imaging system using a Multispectral Filter Array (MSFA) is expected to be widely used in the industrial world because the equipment cost is low and simple imaging is possible. Specifically, a one-shot spectroscopic imaging camera provided with an MSFA on the incident side of the receiver has been proposed (see, for example, Non-Patent Document 1).

On the other hand, in recent years, there has been a demand to measure the polarization characteristics of optical parts, thin film products, and biological tissues, that is, birefringence, and to clarify the detailed physical properties and behavior of the object to be measured based on the polarization characteristics obtained by the measurement. be.

The rotary photon method is a conventional method for measuring the main polarization characteristics. However, in the rotary photon method, it is necessary to prepare a rotary motor, a rotation angle detection sensor, a housing, etc. for rotating the polarizing filter, which complicates the measurement optical system and data on the surface of the measurement surface. There was a problem that misalignment was likely to occur.

Therefore, the polarization state of the transmitted light transmitted from the object to be measured is changed, that is, a light receiving element such as a CCD (Charge Coupled Device) is provided on the incident side with a light receiving element, that is, a polarizer having the same size as the pixel and the same number as the pixel. A one-shot polarized imaging camera has been proposed (see, for example, Non-Patent Document 2).

P-J. Lapray, X. Wang, J-B. Thomas, P. Gouton, “Multispectral Filter Arrays: Recent Advances and Practical Implementation,” Sensors, Vol.14, pp.21626-21659, 2014 Shojiro Kawakami, Takayuki Kawashima, Yoshihiko Inoue, Yo Honma, Takashi Sato, Shinichi Ota, Kiyoshi Nagashima, Takafumi Aoki, "Development of Polarized Imaging Camera Using Photonic Crystal Polarizer", IEICE Transactions C, Vol. .90, No.1, pp.17-24, Jan. 2007

The conventional one-shot spectroscopic imaging camera can acquire MSI by acquiring different spectroscopic information for each pixel in one shot. However, the polarization (that is, polarization orientation) information is limited to acquisition in a polarization state that matches the polarization transmission characteristics of the multispectral filter of each pixel, and the polarization information in all orientations of the object to be measured can be obtained with one shot. You can't get it.

On the other hand, the conventional one-shot polarized light imaging camera can acquire an image for each of the four directions of polarization in one shot by acquiring polarization information in four directions different for each pixel. However, the spectrum (that is, wavelength) information is limited to the acquisition within the wavelength band having the spectral transmission characteristics of the polarizer of each pixel, and the color image and spectroscopy of the object to be measured can be obtained in one shot. It is difficult to get information. Therefore, for example, for medical purposes, a polarizing microscope has been put into practical use for visualizing collagen, but since such a polarizing microscope has a different configuration from a normal optical microscope, it is used for diagnosis and imaging of RGB images or MSI. Separately requires an optical microscope and other optical equipment.

That is, regardless of whether a conventional one-shot spectroscopic imaging camera or a one-shot polarization imaging camera is used, it is difficult to acquire MSI and polarization information at the same time in one shot, and in order to acquire both MSI and polarization information. Has a problem that it takes time and effort because it requires multiple shootings.

Hereinafter, in the present specification, on the measurement surface of a one-shot spectroscopic imaging camera, a one-shot polarized imaging camera, or the like (optical equipment), the smallest unit whose orientation is oriented or the smallest unit having a predetermined spectral transmission characteristic is referred to as a "cell". It is called. The plurality of "cells" include all those that are individually separated, those that are adjacent to each other and have a frame or the like on the outer periphery, and the like.

When acquiring MSI and polarization information at the same time in one shot, it is conceivable that the MSFA and the polarizing filter array should be overlapped in principle.
However, the present inventor has focused on the fact that it is difficult to obtain MSI and polarization information in one shot at the same time and in a good manner even if the conventional multispectral array and the polarizing filter array are simply superposed. Then, the present inventor has found that it is important that the correlation between the spectral transmission characteristic and the polarization transmission characteristic for each cell satisfies a predetermined condition, and has completed the present invention.

The present invention has been made to solve the above-mentioned problems, and provides a filter, an image capturing apparatus, and an imaging system capable of simultaneously acquiring MSI and polarization information in one shot.

Filter according to the present invention includes a plurality and variety of orientations of the type and the polarization of the wavelength band that transmits the spectral polarization cell assigned respectively, the type of orientation of the polarization Ri der 4 or more, the spectroscopic polarimetry cell Consists of a single photonic crystal having a lattice pitch that allows light of the assigned wavelength band type to pass through and a lattice direction that matches the assigned polarization orientation type. It is characterized by that.
Here, there are a plurality of types of wavelength bands of light to be transmitted, and there are also a plurality of types of polarization directions to be transmitted. The types of wavelength bands between adjacent spectrally polarized cells and the types of polarization directions between adjacent spectrally polarized cells may be the same as each other or may be different from each other.

The image capturing apparatus according to the present invention is characterized in that the above-mentioned filter is provided in the light receiving unit.

Image capturing system according to the present invention includes the above image imaging apparatus, respectively the type and kind of orientation of the polarization of the wavelength band of permeable light of the spectral polarization cell, each of the spectral polarization cell A storage device that transmits and stores the light intensity of each of the spectrally polarized cells acquired by the imaging device, and the type of the wavelength band of each of the spectrally polarized cells stored in the storage device and the light. The spectral information for each of the spectrally polarized cells is calculated based on the intensity, and for each of the spectrally polarized cells based on the type of the polarization orientation of each of the spectrally polarized cells stored in the storage device and the light intensity. The calculation device that calculates the polarization information and the spectrum information for each spectrally polarized cell calculated by the calculation device are displayed as an image, and the polarization information for each spectrally polarized cell calculated by the calculation device is displayed as an image. It is characterized in that it is provided with a display device for polarizing.

Image capturing system according to the present invention includes the above image imaging apparatus, the type and the type of orientation of the polarization of the wavelength band of light that are assigned to each of the spectral polarization cell, each of the spectral polarization cell The type of the wavelength band assigned to each of the storage device that transmits and stores the light intensity of each spectrally polarized cell acquired by the imaging device and the spectrally polarized cell stored in the storage device. And the spectral information for each of the spectrally polarized cells is calculated based on the light intensity, and based on the type of polarization orientation assigned to each of the spectrally polarized cells stored in the storage device and the light intensity. An arithmetic device that calculates polarization information for each spectrally polarized cell and spectrum information for each spectrally polarized cell calculated by the arithmetic apparatus are displayed as an image, and each spectrally polarized cell calculated by the arithmetic apparatus is displayed. It is characterized by including a display device that displays polarization information as an image.

In the above-mentioned image imaging system, the type of the wavelength band and the type of the direction of the polarization assigned to each of the spectrally polarized cells may be directly introduced into the arithmetic unit without being stored in the storage device.

According to the filter, the imaging apparatus and the imaging system according to the present invention, MSI and polarization information can be obtained in one shot with the same equipment configuration and equipment cost as the conventional one-shot spectroscopic imaging camera and one-shot polarization imaging camera. Can be acquired at the same time.

It is the schematic of the filter and the image taking apparatus of one Embodiment which concerns on this invention. It is a schematic diagram which shows the filter of the 1st aspect to which this invention is applied. It is a schematic diagram for demonstrating the information acquired by using the filter of one Embodiment which concerns on this invention. It is the schematic which shows the filter of the 1st aspect using a photonic crystal. It is the schematic which shows the structure of the filter of the 2nd aspect to which this invention is applied. It is a schematic diagram for demonstrating the image restoration method using the filter of one Embodiment which concerns on this invention, and the image capturing apparatus, and is the figure which shows the structure of the assumed filter. It is a schematic diagram for demonstrating the image restoration method using the filter and the image capturing apparatus of one Embodiment which concerns on this invention, and is the figure which shows the relationship between the composition of the captured image, and the data acquisition point. It is a schematic diagram for demonstrating the image restoration method using the filter and the image-taking apparatus of one Embodiment which concerns on this invention, and is the figure which shows the restoration method of the image taken in the case of 4-band four-polarized light. It is the schematic which shows the image taking system of the 1st aspect to which this invention is applied. It is the schematic which shows the image taking system of the 2nd aspect to which this invention is applied. It is an original multispectral image in an Example. In the embodiment, it is a one-shot captured image created by simulation. In the example, an RGB image of a multispectral image of four kinds of wavelength bands W in a polarization direction φ = 90 ° is shown. In the example, in the wavelength band W = 650 nm, a linear polarized light intensity image obtained based on the directions φ of four kinds of polarized light is shown.

Hereinafter, embodiments of the filter, image capturing apparatus, and imaging system according to the present invention (hereinafter referred to as the present embodiment) will be described with reference to the drawings. The drawings used in the following description are schematic, and the length, width, thickness ratio, etc. are not always the same as the actual ones and can be changed as appropriate.

As shown in FIG. 1, the image capturing device 2 of the embodiment to which the present invention is applied is not particularly limited as long as it includes a light receiving unit 4 and can perform imaging, but for example, a camera, an optical device for imaging, or the like. Is. The light receiving unit 4 is provided with a light receiving element 5 having a plurality of pixels P. The multispectral polarizing filter array (filter, hereinafter referred to as a filter array) 6A of the present embodiment is provided on the light receiving element 5 on the side where light is incident.

The filter array 6A shown in FIG. 1 is the first aspect of the filter to which the present invention is applied, in which a plurality of spectrally polarized cells C1, C2, ..., Cn are arranged. The plurality of spectrally polarized cells C1, C2, ..., Cn are respectively in the pixels P1, P2, ..., Pq of the light receiving element 5 (however, in FIG. 1, the pixels P1, P2 are hidden in the filter array 6A). It is formed so that it can be arranged facing each other. The number n of the plurality of spectrally polarized cells C is appropriately set according to the number q of the pixels P of the light receiving element 5 so that the filter array 6A can face all the pixels P of the light receiving element 5. .. As shown in FIG. 1, when one pixel P faces one spectrally polarized cell C, n = q.

In this specification, the number of spectrally polarized cells C, the parameters related to the spectrally polarized cells C, and the number of components that can face the spectrally polarized cells C will be described as n. Further, in terms of the spectrally polarized cell C, the first spectrally polarized cell C is represented by C1, the second spectrally polarized cell C is represented by C2, ..., And the unspecified x, yth spectrally polarized cell C is Cx, Cy. The same expression is applied to the parameters related to the spectrally polarized cell C and the components that can face the spectrally polarized cell C. Although not shown, each spectral polarization cell Cx is assigned a specific wavelength band Wx and a polarization direction φx, transmits light having a wavelength within the wavelength band Wx, and transmits light having an oriented direction φx. It can be made transparent.

The types of wavelength bands Wx and Wy assigned to the adjacent spectrally polarized cells Cx and Cy are basically different from each other. The meaning of "basically" indicates that adjacent spectrally polarized cells Cx and Cy should be able to transmit each other in a specific wavelength band Wx and Wy. For example, the spectral polarization cell Cx is assigned a wavelength band of 400 nm or more and 450 nm, and the spectral polarization cell Cy is assigned a wavelength band of 450 nm or more and less than 500 nm. However, if the polarization directions φ described later are assigned differently from each other, a wavelength band of 450 nm or more and less than 500 nm may be assigned to both the spectral polarization cells Cx and Cy.
As described above, the filter array 6A is a plate of the filter array 6A instead of changing the wavelength of the light to be passed by exchanging the filter through which the light of the wavelength within each of the wavelength bands Wx and Wy is passed. The surface (that is, the light receiving surface of the light receiving unit 4 of the image capturing device 2) is spatially divided so that spectral information of a plurality of wavelengths can be acquired in one shot.

The types of polarization directions φx and φy assigned to the adjacent spectral polarization cells Cx and Cy are also different from each other.
FIG. 2 compares and compares the information of the light received by the pixel P facing the four spectrally polarized cells C adjacent to each other in the vertical direction and the horizontal direction (that is, each direction of the row and column of the filter array 6A) on the paper surface. A configuration example is shown in which a calculation is performed to acquire polarization information and spectroscopic information of a measurement object (not shown). Therefore, for example, the polarization directions φx, φx + 1, φy, φy + 1 assigned to each of the four spectral polarization cells Cx, Cx + 1, Cy, and Cy + 1 are set to −45 °, 0 °, 45 °, and 90 °. .. That is, the filter array 6A spatially divides the plate surface of the filter array 6A (that is, the light receiving surface of the image capturing apparatus 2) instead of changing the direction of the light passing through by rotating the polarizing element, and a plurality of the filter array 6A. It is also possible to acquire the polarization information of the direction in one shot.

For example, in the filter array 6A, when there are m types of wavelength bands W, k types of polarization directions φ, and any of these types is assigned to a plurality of spectrally polarized cells C, it is shown in FIG. In the configuration example, m = 16, k = 4. As another example, in the case of m = 6, k = 4 (the types of polarization directions φ are four types of −45 °, 0 °, 45 °, and 90 °), as shown in FIG. Spectral information of 6 types of wavelength bands W can be obtained for 4 types of polarized light having an azimuth φ.
The number of types of wavelength band W and the number of types of polarization direction φ may or may not match as in the above example. In any case, if the type of the wavelength band W assigned to each of the spectroscopic polarization cells C and the type of the polarization direction φ are known, the wavelength band W and the polarization direction can be used by using the processing method or the like described later. Spectral information and polarization information for each spectroscopic polarization cell C based on φ can be obtained.

The filter array 6A is particularly limited as long as it has a plurality of spectrally polarized cells C and it is possible to assign the type of wavelength band W and the type of polarization direction φ to each spectrally polarized cell C. Not limited. Examples of those constituting such a filter array 6A include a photonic crystal and the like.

A photonic crystal is a device composed of a multilayer interference film as shown in FIG. 4 and having excellent transmittance. When each spectrally polarized cell C is composed of a photonic crystal, the spectral transmission characteristic, that is, the transmission wavelength band can be changed by changing the lattice pitch for each spectrally polarized cell C. In other words, the lattice pitch of the photonic crystal constituting the spectrally polarized cell C is appropriately set according to the type of the wavelength band W through which the spectrally polarized cell C is transmitted.

The photonic crystal is also a device capable of controlling the polarization characteristics depending on the lattice direction. In other words, the lattice directions of the photonic crystals constituting each spectroscopic polarization cell C are appropriately set according to the type of polarization direction φ passing through the spectral polarization cell C.
Therefore, as shown in FIG. 4, in the multilayer interference film formed on the substrate made of quartz or the like, the light transmitted to each spectrally polarized cell C is transmitted by changing the lattice direction and the interval for each spectrally polarized cell C. The spectral characteristics and polarization characteristics can be easily adjusted. As described above, when the photonic crystal is used, the filter array 6A by a single device is realized. The plane size of one spectrally polarized cell C is, for example, 5 μm × 5 μm. Examples of the multilayer interference film include a laminated film of _{niobium dioxide (Nb 2} O ₂ ) and silicon dioxide (SiO _2). The lattice pitch is set to about 300 nm or less, but these values are not particularly limited.

According to the filter array 6A of the first aspect described above, when light containing spectral information and polarization information is incident, such as light from a measurement object (not shown), the spectral polarization cells C1, C2, ..., In each of the Cn, light within the assigned predetermined wavelength bands W1, W2, ..., Wn and in the predetermined polarization directions φ1, φ2,…, φn can be transmitted in parallel for each spectrally polarized cell C. can. Further, by using a photonic crystal, light in a wavelength band W1, W2, ..., Wn and having a predetermined polarization direction φ1, φ2, ..., Φn can be spectrally polarized by a single device. Each C can be transmitted in parallel.

A second aspect of the filter to which the present invention is applied is a filter array 6B. In the filter array 6B, a plurality of composite cells c1, c2, ..., Cn are arranged in place of the spectrally polarized cell C of the filter array 6A. The plurality of composite cells c1, c2, ..., Cn are formed so as to be able to face each of the pixels P1, P2, ..., Pq of the light receiving unit 4 of the image capturing device 2 as shown in FIG. ..

The filter array 6B integrally includes the MSFA (spectral filter array) 11 shown in FIG. 5 and the polarizing filter array 12, and specifically, the MSFA 11 and the polarizing filter array 12 are laminated.

In MSFA11, a plurality of spectroscopic cells SC1, SC2, ..., SCn to which the type of wavelength band W is assigned are arranged. On the other hand, in the polarization filter array 12, a plurality of polarization cells PC1, PC2, ..., PCn to which the type of polarization direction φ is assigned are arranged.
That is, the composite cells c1, c2, ..., Cn of the filter array 6B are obtained by stacking the spectroscopic cells SC1, SC2, ..., SCn and the polarizing cells PC1, PC2, ..., PCn, respectively. As long as it does not affect the optical characteristics of the transmitted light, there may be one interposed between the MSFA 11 and the polarizing filter array 12.

However, as conditions for overlapping the MSFA 11 and the polarizing filter array 12, it is important that the polarization characteristics of the MSFA 11 are sufficiently weak and that the spectral characteristics of the polarizing filter array 12 have sufficient sensitivity in the measurement target range.

That is, the polarization directions transmitted by the spectroscopic cells SC1, SC2, ..., SCn are at least equal to the polarization directions φ1, φ2, ..., φn transmitted by the polarized cells PC1, PC2, ... It suffices to include the directions φ1, φ2, ..., Φn. On the other hand, the wavelength band of the light transmitted by the polarizing cells PC1, PC2, ..., PCn is at least one in the wavelength bands W1, W2, ..., Wn of the light transmitted by the spectroscopic cells SC1, SC2, ... However, the wavelength bands W1, W2, ..., Wn may be included. Specifically, as illustrated in FIG. 5, the wavelength band W'of light transmitted by the polarizing cell PCx includes at least the wavelength band W of light transmitted by the spectroscopic cell SCx. Further, the half-value width ω (PCx) of the wavelength band W'is set to be equal to or larger than the half-value width ω (SCx) of the wavelength band W.

When the above conditions are satisfied, the light transmitted through each of the spectral cells SC1, SC2, ..., SCn is not blocked by the spectral characteristics of the polarizing cells PC1, PC2, ..., PCn, and the polarized cells PC1, The light transmitted through each of the PC2, ..., PCn is not blocked by the polarization characteristics of the spectroscopic cells SC1, SC2, ..., SCn.

FIG. 6 shows an example of a filter array 6B with m = 4 and k = 4. In the MSFA11, the type of wavelength band W to be transmitted differs for each adjacent spectroscopic cell SC, and the structure is the same as when four types of filters are used. The type and configuration of the polarization direction φ are as described in the filter array 6A.

Regarding the sensitivity and placement optimization of MSFA11, known literature (for example, Yudai Yanagi, Kazuma Shinoda, Madoka Hasegawa, Shigeo Kato, Masahiro Ishikawa, Hideki Komagata, Naoki Kobayashi: "Multispectrum considering observation wavelength and filter placement" Filter Array Optimization Method ”, Journal of the Institute of Electronics, Information and Communication Engineers, Vol.J99-D, No.8, pp.794-804, Aug.2016, etc.). In the illustrated known literature, it is disclosed that some original image is prepared, and the image is photographed and restored by the filter array on the simulation. At that time, it is also disclosed that the image restoration is performed by brute force the spectral sensitivity of the MSFA 11 as much as possible, and the filter design is performed by selecting the spectral sensitivity having the highest restoration accuracy.

The MSFA 11 is not particularly limited as long as it has a plurality of spectroscopic cell SCs and the type of wavelength band W can be assigned to each spectroscopic cell SC. Examples of those constituting such MSFA11 include dyes, pigment dyes (hereinafter, may be simply referred to as pigments), synthetic resins, multilayer interference films and the like.

In particular, since the dye does not have a specific polarization characteristic, it is unlikely to block the light transmitted through any of the polarizing cells PC1, PC2, ..., PCn, and is suitable for use in the MSFA11.
Pigments are mainly used in MSFA, which is currently the mainstream, but pigments have an action of canceling polarized light (distortion property). Therefore, for pigments, based on the above conditions, considering the type of polarization direction φ of the light transmitted through the polarizing cells PC1, PC2, ..., PCn, the depolarization property is suppressed by miniaturizing the particles. It is preferably used for MSFA11.
As for the synthetic resin and the multilayer interference film, as described above, the orientations φ1, φ2, ... It is preferable to set various parameters so as to transmit the polarized light of φn without blocking it.

Regarding the sensitivity and arrangement optimization of the polarizing filter array 12, commercially available filters are generally used to acquire information on the orientations of four types of polarized light in a 2 × 2 composite cell. In particular, the original light intensity can be restored by adding the image at the orientation φ = 0 ° and the image at φ = 90 °. It is preferable that four or more types of polarization directions φ are secured, such as 0 °, 90 °, 45 °, and −45 °.

The polarizing filter array 12 is not particularly limited as long as it has a plurality of polarizing cell PCs and the type of polarization direction φ can be assigned to each polarizing cell PC. Examples of those constituting such a polarizing filter array 12 include a photonic crystal, a wire grid structure, a synthetic resin, and a multilayer interference film.

When each polarization cell PC is composed of a photonic crystal, as shown in FIG. 5, the polarization characteristics are controlled by changing the lattice direction according to the type of polarization direction φ assigned to each polarization cell PC. can do. In this case, the lattice pitch of the photonic crystals constituting the respective polarizing cell PCs does not block at least the light of the wavelength bands W1, W2, ..., Wn transmitted through each of the opposing spectroscopic cells SC1, SC2, ..., SCn. , It is set uniformly so that it is transparent.

The wire grid structure is, for example, a device sandwiched between glass substrates made of fused quartz or the like and composed of a plurality of metal wires arranged in parallel with each other. The wire grid structure originally has a wide transmission wavelength band. For example, in the case of a visible region wire grid polarizer, an antireflection coating corresponding to 400 nm to 700 nm may be applied, and all types of wavelength bands W can be set as transmission wavelength bands.

Further, the wire grid structure is a device capable of controlling the polarization characteristics depending on the direction in which the metal wire is stretched. Therefore, when each polarized cell PC is configured with a wire grid structure, the direction in which the metal wire is stretched matches the direction φ of the polarized light assigned to each polarized cell PC. In other words, the grid direction of the wire grid structure constituting the polarized cell PC is appropriately set according to the direction φ of the polarized light transmitted through the polarized cell PC.

According to the filter array 6B of the second aspect described above, when light containing information on a measurement object (not shown) is incident, the wavelengths of the assigned types are assigned in each of the spectroscopic cells SC1, SC2, ..., SCn. Light in the bands W1, W2, ..., Wn can be collectively transmitted in parallel. Further, in each of the polarized cells PC1, PC2, ..., PCn that can face each of the spectroscopic cells SC1, SC2, ..., SCn, the light of the assigned types of directions φ1, φ2, ..., Φn is collectively collected in parallel. Can be made transparent. Therefore, according to the filter array 6B, the same effect as that of the filter array 6A can be obtained.

Further, according to the image capturing apparatus 2 provided with the filter arrays 6A and 6B, when light or the like containing information on the object to be measured is received, the spectroscopic polarization cell C or the spectroscopic cell facing each of the plurality of pixels P The optical information of the object to be measured in the wavelength band W and the polarization direction φ assigned to the SC and the polarization cell PC can be collectively acquired in one shot. For example, each pixel P of the light receiving element 5 acquires a light intensity that reflects the above-mentioned optical information.

Here, an example of an image taken by the image capturing device 2 provided with the filter array 6A and a method of restoring various images from the image will be described. Needless to say, the same image restoration method as that of the image capturing device 2 provided with the filter array 6A can be applied to the image capturing device 2 provided with the filter array 6B.

As an example, assume an image taken by the filter array 6A configured as shown in FIG. The type of wavelength band W assigned to each of the spectrally polarized cells C1, C2, ..., Cn of the filter array 6A is 450 nm (“1” shown in FIG. 6, Band 1 shown in FIG. 7) and 550 nm (shown in FIG. 6). There are four types: "2", Band 2 shown in FIG. 7, 650 nm ("3" shown in FIG. 6, Band 3 shown in FIG. 7), and 750 nm ("4" shown in FIG. 6, Band 4 shown in FIG. 7). .. On the other hand, the types of polarization directions φ assigned to each of the spectral polarization cells C1, C2, ..., And Cn of the filter array 6A are −45 ° (“\” shown in FIG. 6) and 0 ° (shown in FIG. 6). There are four types: “−”), 45 ° (“/” shown in FIG. 6), and 90 ° (“|” shown in FIG. 6).

The image taken by using the filter array 6A shown in FIG. 6 is an image having the meaning shown in FIG. For example, among the types of wavelength band W, for 450 nm, the four spectrally polarized cells C shown in the range surrounded by the thick broken line (counting from the upper left to the right side of the paper, and counting continuously in the next row) can be said. For example, the spectroscopic polarization cells C1, C3, C9, C11) receive light containing information on the object to be measured. On the other hand, among the types of polarization direction φ, for 0 °, the four spectral polarization cells C shown in the range surrounded by the thin broken line (counting from the upper left to the right side of the paper, and counting continuously in the next row). Speaking of which, the spectrally polarized light cells C1, C7, C10, C16) receive light including information on the object to be measured.

As shown in FIG. 7, the illustrated filter array 6A can acquire an image based on four kinds of wavelength bands W and four kinds of polarization directions φ. However, since the spectrally polarized light cells C to which the type of the specific wavelength band W and the type of the specific polarization direction φ are assigned are separated from each other, when the light from the measurement target is actually received, the specific wavelength is specified. Data is missing from the pixels (white pixels shown in FIG. 8) P other than the spectroscopic polarization cell C that can acquire the type of the band W and the type of the polarization direction φ. Therefore, it is considered that the restoration process is necessary in practice, but the existing pixel interpolation method can be applied as it is to the restoration process. As the simplest method for such a restoration process, as shown in FIG. 8, although the resolution is substantially reduced, the light intensity (that is, optical information) received by the uninterpolated spectrally polarized cell C is deleted and observed. An example is a method of collecting the light intensities received only by the target spectrally polarized cell C.

Further, when complementing the captured image, conventional pixel interpolation methods such as nearest neighbor interpolation, interpolation interpolation, and least squares method can be applied as they are. Since several pixel interpolation methods for multispectral images have been proposed so far, it is also possible to apply these pixel interpolation methods to information acquisition in the spectroscopic polarization cell C to which the type of polarization direction φ is assigned. can do.

Further, another example of a method of restoring various images from an image taken by the image capturing device 2 provided with the filter array 6A will be described.
Assuming that the column vector representation of the light incident on the filter array is x, the sensitivity matrix of the filter array is Φ, and the captured image is y, the captured image y is the formula y = Φx (that is, the calculation formula by the matrix). Can be modeled.

According to the above modeling, the restoration of the incident light is to obtain x that minimizes | y−Φx |, assuming that the captured image y is known. Therefore, the simplest restoration method is to obtain the estimated value of x by the relational expression of ^{x ^ = Φ + y, using Φ +} as the generalized pseudo-inverse matrix. In this specification, the estimated value of x is referred to as “x ^”.

It is also possible to obtain x ^ by independently bilinear interpolating the images of each type of polarization direction φ in each type of wavelength band W. Since x ^ is a spectropolarized image (that is, the image shown in FIG. 3), each image can be calculated as follows. The following method is an example of a method of calculating various images from a spectropolarized image, and is not limited to the following contents.

For example, a spectroscopic image is obtained by adding an image with an orientation of 90 ° (or an image with an orientation of 45 ° and an image with −45 °) to an image with an orientation of 0 ° of x ^.
For RGB images, if the spectral image is r, the diagonal matrix of illumination light intensity is E, the XYZ color matching function is T, and the XYZ-RGB color transformation matrix is C, then the linear RGB image g is g = It is obtained by a matrix calculation called CTER. After such matrix calculation, the obtained linear RGB image g is subjected to appropriate color gamut conversion and gamma correction according to the display device and the reproduction device to obtain an RGB image.
Further, the linearly polarized light intensity image is obtained in the image of each wavelength band of x ^ by the following calculation:
{(Image with 0 ° direction-Image with 90 ° direction) ² + (Image with 45 ° direction-Image with -45 ° direction) ² } / (Image with 0 ° direction + Image with 90 ° direction),
Obtained by executing.

Next, an image capturing system according to an embodiment to which the present invention is applied will be described.
As shown in FIG. 9, the image capturing system 20A of the first aspect includes an image capturing device 2 having a filter array 6A in the light receiving unit 4, a storage device 22A, an arithmetic unit 24A, and a display device 26. I have.

The storage device 22A is a so-called memory, and includes the type of the wavelength band W of light assigned to each of the spectroscopic polarization cells C1, C2, ..., Cn, the type of the polarization direction φ, and the spectral polarization cells C1, C2. It is a device that transmits each of ..., Cn and stores the light intensity for each of the spectrally polarized cells C1, C2, ..., Cn acquired by the image capturing device 2.
The storage device 22A may be provided outside the image capturing device 2 as shown in FIG. 9, or may be built in the image capturing device 2 although not shown.

The arithmetic device 24A is based on the type of wavelength band W assigned to each of the spectrally polarized light cells C1, C2, ..., Cn and the light intensity acquired by the image capturing device 2 stored in the storage device 22A. Spectral information for each spectrally polarized cell C1, C2, ..., Cn is calculated. Further, the arithmetic unit 24A determines the type of polarization direction φ assigned to each of the spectrally polarized light cells C1, C2, ..., Cn and the light intensity acquired by the image capturing device 2 stored in the storage device 22A. Based on this, the polarization information for each of the spectrally polarized light cells C1, C2, ..., Cn is calculated.
The arithmetic unit 24A may also be provided outside the image capturing device 2 as shown in FIG. 9, or may be built in the image capturing device 2 although not shown.

In the arithmetic unit 24A, spectroscopic information and polarization information for each of the spectral polarization cells C1, C2, ..., Cn can be calculated by using the above-mentioned image restoration method, known image restoration method, and complementary method.

The display device 26 can be said to be a reproduction device, and displays spectral information for each of the spectrally polarized light cells C1, C2, ..., Cn calculated by the arithmetic unit 24A as an image, and the spectrally polarized light calculated by the arithmetic unit 24A. Polarization information for each of cells C1, C2, ..., Cn is displayed as an image.
The display device 26 may be provided outside the image capturing device 2 as shown in FIG. 9, or may be attached to the exterior body of the image capturing device 2 although not shown.

In the imaging system 20A of the first aspect described above, physical property data such as the type of wavelength band W assigned to each of the spectrally polarized light cells C1, C2, ..., And Cn and the type of polarization direction φ are stored in the storage device 22A. Each type of wavelength band W and polarization direction φ included in the light intensity data is utilized in the arithmetic unit 24A.

According to the imaging system 20A of the first aspect, one is based on the type of wavelength band W and the type of polarization direction φ assigned to each of the spectrally polarized light cells C1, C2, ..., Cn of the filter array 6A. The polarization information and the spectral information of the measurement object B can be simultaneously acquired by the shot, and the acquired polarization information and the spectral information of the measurement object can be displayed as an image (see FIG. 3). Therefore, it is possible to instantly and easily show the polarization information and the spectral information of the measurement object to the user of the image capturing system 20A.

As shown in FIG. 9, as a modification of the image capturing system 20A of the first aspect, the image capturing device 2 provided with the filter array 6B in the light receiving unit 4, the storage device 22A, and the arithmetic unit 24A are displayed. An image capturing system 20B having the device 26 and the device 26 can be mentioned. The image capturing system 20B may be applied by replacing the spectrally polarized cells C1, C2, ..., Cn with the composite cells c1, c2, ..., Cn in the description of the image capturing system 20A, and thus the description thereof will be omitted.

As shown in FIG. 10, the image capturing system 20C of the second aspect includes an image capturing device 2 having a filter array 6A in the light receiving unit 4, a storage device 22C, an arithmetic unit 24C, and a display device 26. I have. Since the image capturing device 2 and the display device 26 are the same as the image capturing device 2 and the display device 26 of the image capturing system 20A, the description thereof will be omitted.

The storage device 22C is a so-called memory, and transmits each of the spectrally polarized cells C1, C2, ..., Cn, and only the light intensity of each of the spectrally polarized cells C1, C2, ..., Cn acquired by the imaging device 2. Is a device that stores so-called metadata.
The storage device 22C may be provided outside the image capturing device 2 as shown in FIG. 10, or may be built in the image capturing device 2 although not shown.

In the arithmetic unit 24C, the type of the wavelength band W and the type of the polarization direction φ assigned to each of the spectral polarization cells C1, C2, ..., And Cn of the filter array 6A are introduced. The type of the wavelength band W and the type of the polarization direction φ may be input in advance to the arithmetic unit 24C, or may be directly input to the arithmetic unit 24C from the outside of the image capturing system 20C.
The arithmetic unit 24C is based on the introduced spectral polarization cells C1, C2, ..., The type of wavelength band W for each Cn, and the light intensity stored in the storage device 22A, and the spectral polarization cells C1, C2, ..., The spectrum information for each Cn is calculated. Further, the arithmetic unit 24C is spectroscopically polarized light based on the type of polarization direction φ assigned to each of the introduced spectrally polarized light cells C1, C2, ..., Cn and the light intensity stored in the storage device 22A. Polarization information for each of cells C1, C2, ..., Cn is calculated.
The arithmetic unit 24C may also be provided outside the image capturing device 2 as shown in FIG. 10, or may be built in the image capturing device 2 although not shown.

In the imaging system 20C of the second aspect described above, the light intensity data acquired in each of the spectrally polarized cells C1, C2, ..., Cn is stored in the storage device 22A, and the spectrally polarized cells C1, C2, ... After each type of wavelength band W and polarization direction φ assigned to each of ， Cn is obtained by the arithmetic unit 24A, the light intensity data is utilized.
Although the flow of data acquisition and utilization is different as described above, according to the image capturing system 20C of the second aspect, the same effect as that of the image capturing system 20A can be obtained.

As shown in FIG. 10, the image capturing system 20C of the second aspect is also modified as an image capturing device 2, a storage device 22C, and an arithmetic unit 24C having a filter array 6B in the light receiving unit 4. An image capturing system 20D having the display device 26 and the display device 26. The image capturing system 20D may be applied by replacing the spectrally polarized cells C1, C2, ..., Cn with the composite cells c1, c2, ..., Cn in the description of the image capturing system 20C, and thus the description thereof will be omitted.

As described above, according to the filter, the imaging apparatus and the imaging system to which the present invention is applied, MSI and polarization are carried out with the same equipment configuration and equipment cost as the conventional one-shot spectroscopic imaging camera and one-shot polarized imaging camera. Information and information can be acquired at the same time in one shot.
Further, according to the filter, the image capturing apparatus and the imaging system to which the present invention is applied, the acquired multispectral polarized image, that is, the multispectral image including the spectral information and the polarization information of the measurement object, does not have the polarization information. Since a normal multispectral image, an RGB image, and a polarization intensity image can be obtained, the roles of the conventional RGB image sensor, multispectral image sensor, and polarized image sensor can be integrated into one sensor.

Although the preferred embodiments of the present invention have been described in detail above, the present invention is not limited to the specific embodiments, and various aspects of the present invention are described within the scope of the claims. It can be transformed and changed.

For example, FIG. 1 and the like exemplify filter arrays 6A and 6B in which various cells (that is, spectroscopic polarized cells C, spectroscopic cells SC, and polarized cells PC) are arranged so as to be adjacent to each other in each direction of rows and columns. As described above, the various cells of the filter arrays 6A and 6B do not necessarily have to be adjacent to each other, and may not be arranged. That is, the various cells of the filter arrays 6A and 6B may be arranged at arbitrary relative positions as long as the wavelength band W assigned to each cell and the polarization direction φ are known.

Further, the type of the wavelength band W assigned to the spectroscopic polarization cell C, the spectroscopic cell SC, and the polarization cell PC, the type of the polarization direction φ, and the number of each type are not limited to the types and numbers described in the above-described embodiment. For example, when there are four types of polarization directions φ, there may be four types other than 0 °, 45 °, 90 °, and −45 °. The number of types of polarization directions φ is not limited to four, and may be three or less, or five or more.
Further, each of the plurality of various cells may be configured in a tile shape so as to be able to face each other over the plurality of pixels P of the light receiving element 5.

Next, an example performed to support the effects of the filter, the image capturing device, and the image capturing system according to the embodiment of the present invention will be described. The present invention is not limited to the following examples.

Multispectral images of four types of wavelength bands W and four types of polarized light directions φ were photographed and restored. Two halogen light floodlights (model number: CHP-500-0.3, size name: 500W, manufacturer: Caster) were used as the light source. The objects to be measured were a commercially available colored pencil, a metal stapler, a pottery figurine and a pot, and a mobile terminal having an LCD screen placed side by side. There were four types of wavelength bands W: 450 nm, 550 nm, 650 nm, and 750 nm. In addition, there are four types of polarization azimuth φ, −45 °, 0 °, 45 °, 90 °, and a polarizing plate (model number: TEG1465DU, manufacturer: Nitto Denko KK) was used to change the azimuth φ. .. In addition, a hyperspectral camera (model number: NH-7, number of pixels: 1280 × 1024 pixels, manufacturer: Eva Japan Co., Ltd.) was used to photograph the object to be measured and to disperse the wavelength band W.

In this embodiment, a multispectral image was taken four times using a multispectral camera while changing the angle of the polarizing plate (that is, the type of the orientation φ). The captured multispectral image was used as the original, and a grayscale image was acquired by thinning out with a filter array on the simulation. Then, by interpolation interpolation, a so-called 4-band 4-polarization azimuth multispectral image was restored, and an RGB image and a linearly polarized light intensity image were acquired from the restored image.

FIG. 11 shows an RGB image of a multispectral image at φ = 90 ° among the original multispectral images with 4 bands and 4 polarization directions. The original image shown in FIG. 11 is originally a full-color image, but is displayed in gray scale in this drawing. FIG. 12 shows a one-shot captured image created by the simulation as described above. FIG. 13 shows an RGB image of a multispectral image of four types of wavelength bands W (that is, four bands) at φ = 90 ° as an example of the four types of polarization directions φ. Further, FIG. 14 shows, as an example of the four types of wavelength bands W, obtained linearly polarized light intensity images based on the directions φ of the four types of polarized light at W = 650 nm.

As shown in FIGS. 11 to 14, it can be seen that the RGB image in the specific polarization direction φ and the polarization intensity image in the specific wavelength band W are restored from the original multispectral image. In particular, in the polarization intensity image shown in FIG. 14, a mobile terminal screen having strong polarization characteristics and a stapler having strong gloss are displayed in white, and a good polarization intensity image is restored.

From the results of the examples described above, by realizing the filter arrays 6A and 6B to which the present invention is applied, the wavelength bands assigned to each of the various cells (that is, the spectroscopic polarization cell C, the spectroscopic cell SC, and the polarization cell PC) are assigned. It can be said that the light of the type W and the type of the predetermined polarization direction φ can be collectively transmitted to each of the various cells, and the spectral information and the polarization information can be acquired at the same time in one shot.

According to the filter, the image capturing apparatus and the imaging system to which the present invention is applied, it is possible to acquire the spectral information and the polarization information at the same time in one shot as described above. The actual implementation of the filter to which the present invention is applied can be realized by combining existing technologies. In addition, restoration of a multispectral polarized image from data acquired using a filter, an image capturing device and an imaging system to which the present invention is applied, and an RGB image, a multispectral image, and a polarization intensity from the obtained multispectral polarized image. Calculation of images and the like can also be realized by existing technology. As described above, the filter, the imaging apparatus and the imaging system to which the present invention is applied have various advantages, and are expected to be widely used in various industrial fields that require acquisition of spectroscopic information and polarization information.

2 ... Imaging devices 6A, 6B ... Filter array (filter)
20A, 20B, 20C, 20D ... Imaging system 22A, 22C ... Storage device 24A, 24C ... Arithmetic device 26 ... Display device C ... Spectral polarization cell c ... Composite cell PC ... Polarization cell SC ... Spectroscopic cell W, W1, W2 ..., Wn ... Wavelength band φ, φ1, φ2, ..., φn ... Polarization direction

Claims (4)

It is equipped with a plurality of spectrally polarized cells to which the type of wavelength band to be transmitted and the type of polarization direction are assigned to each.
The type of orientation of polarization Ri der 4 or more,
The spectrally polarized cell is a single photonic crystal having a lattice pitch that allows light of the assigned wavelength band type to pass through and a lattice direction that matches the allocated polarization orientation type. It is configured,
filter. The filter according to claim 1 is provided in the light receiving unit.
Imaging device. And images capturing apparatus according to請Motomeko 2,
The type of the wavelength band of light assigned to each of the spectrally polarized cells, the type of the direction of the polarized light, and each of the spectrally polarized cells transmitted through each of the spectrally polarized cells and acquired by the imaging apparatus. A storage device that stores light intensity and
The spectral information for each spectrally polarized cell is calculated based on the type of the wavelength band assigned to each of the spectrally polarized cells stored in the storage device and the light intensity, and the said stored in the storage device. An arithmetic device that calculates polarization information for each spectrally polarized cell based on the type of polarization orientation assigned to each of the spectrally polarized cells and the light intensity.
A display device that displays the spectral information for each spectrally polarized cell calculated by the arithmetic unit as an image, and displays the polarization information for each spectrally polarized cell calculated by the arithmetic unit as an image.
To prepare
Imaging system. And images capturing apparatus according to請Motomeko 2,
A storage device that transmits light through each of the spectrally polarized cells and stores the light intensity of each of the spectrally polarized cells acquired by the imaging device.
The spectral information for each spectrally polarized cell is calculated based on the type of the wavelength band assigned to each of the spectrally polarized cells and the light intensity stored in the storage device, and is assigned to each of the spectrally polarized cells. An arithmetic device that calculates polarization information for each spectroscopic polarization cell based on the type of polarization orientation and the light intensity stored in the storage device.
A display device that displays the spectral information for each spectrally polarized cell calculated by the arithmetic unit as an image, and displays the polarization information for each spectrally polarized cell calculated by the arithmetic unit as an image.
To prepare
Imaging system.

Images