JPWO2016031922A1 - Multispectral camera - Google Patents

Abstract

The multispectral camera has first to Nth (N is an integer of 2 or more) optical filters having different total peak numbers including the peak number of transmittance and the number of underpeaks. In relation to each transmittance Y, the transmittance Y periodically changes with respect to a change in the variable Z related to the wavelength λ.

Description

  The present invention relates to a multispectral technique capable of measuring reflection spectral characteristics of a subject in units of pixels from an image acquired by imaging the subject.

  In recent years, there has been a remarkable increase in functionality and performance of digital cameras using image sensors such as CCDs and CMOSs. In particular, due to rapid progress in semiconductor manufacturing technology, the pixel structure in an image sensor has been miniaturized. As a result, the number of pixels of the image sensor and the high integration of the drive circuit are improved, and the image quality of the image sensor is greatly improved. In addition, the miniaturization of the image sensor has been realized.

  Recently, color image quality has improved significantly. In addition to color imaging using the three primary color filters of normal red (R), green (G), and blue (B), imaging using a number of color filters improves the quality of the color image of the subject. Things have been done. For example, the multispectral camera introduced in Patent Document 1 is equipped with a number of optical filters having different spectral characteristics in front of the CCD, and images are taken while mechanically replacing each optical filter. Thereby, detailed color information and spectrum information of the subject can be obtained from each image acquired by imaging.

  Patent Document 2 introduces a multispectral camera that can reduce the number of optical filters to be used by optimally designing the optical filters.

  Patent Document 3 discloses a mechanism for replacing an optical filter by using a one-dimensional imaging device, a beam splitter that splits light incident through a lens, and a tunable filter that selectively transmits light in an arbitrary wavelength band. This paper introduces technologies that can obtain images in various wavelength bands without using.

  Further, Patent Document 4 introduces a technique for obtaining a color image with good quality using a single-plate color image pickup device including standard RGB color filters and other color filters.

JP 2005-181038 A JP 2010-122080 A JP 2012-138852 A JP 2008-136251 A

  In the prior art disclosed in Patent Document 1 and Patent Document 2, a mechanical mechanism for replacing a plurality of optical filters is required. For this reason, there is a problem that the size of the image pickup apparatus is increased, and the mechanical part must be regularly maintained. In the prior art disclosed in Patent Document 3, since a one-dimensional image sensor is used, the one-dimensional image sensor is moved in a direction perpendicular to the pixel arrangement direction in order to obtain a two-dimensional image (hereinafter referred to as scan). A mechanical mechanism is also required. Furthermore, if the subject is not stationary, there is a problem that the imaging result becomes a two-dimensional image having blur. In the prior art disclosed in Patent Document 4, the standard RGB color filters used and the other color filters transmit only light in one specific (narrow) wavelength band. There is a problem that further improvement in sensitivity cannot be expected.

  The present invention provides a multi-spectral technique with a different approach than the prior art. An embodiment of the present invention provides a multispectral camera that does not require a mechanical mechanism, can cope with movement of a subject, and can improve sensitivity.

  A multispectral camera according to an aspect of the present invention is an imaging element in which a plurality of pixel units are two-dimensionally arranged on an imaging surface, and each pixel unit is from 1 to N (N is an integer of 2 or more). Each of the light sensing cells outputs a photoelectric conversion signal corresponding to the amount of light received, and is arranged to face the imaging element and the first to Nth light sensing cells, respectively. N optical filters, each having one or more transmittance peaks or underpeaks in a predetermined wavelength band, and the total number of peaks combined with the number of peaks and the number of underpeaks is different. 1 to Nth optical filter, and a signal processing circuit for processing photoelectric conversion signals output from the first to Nth photosensitive cells. The transmittance Y of each of the first to Nth optical filters changes periodically with respect to the change of the variable Z associated with the wavelength λ in the predetermined wavelength band. The signal processing circuit includes N photoelectric conversion signals from the first to Nth photosensitive cells, N types of periodic functions having the independent variable as the variable Z, and the variable for the minute change in the wavelength λ. The spectral characteristic of the captured image is calculated using the change value of Z.

  A multispectral camera according to another aspect of the present invention includes a compound eye optical system including N + 1 (N is an integer of 2 or more) optical systems and one or more transmittance peaks or underpeaks in a predetermined wavelength band. In addition, the first to Nth optical filters having different total peak numbers including the number of the peaks and the number of the underpeaks, and the first to Nth optical filters facing the first to Nth optical filters, respectively. Each of the photosensitive cells outputs a photoelectric conversion signal corresponding to the amount of received light, and processes the photoelectric conversion signals output from the first to Nth photosensitive cells. A signal processing circuit. The transmittance Y of each of the first to Nth optical filters changes periodically with respect to the change of the variable Z related to the wavelength λ in the predetermined wavelength band, and the first to Nth optical filters The optical filter is disposed to face each of N optical systems of the N + 1 optical systems in the compound eye optical system, and the imaging element further includes any of the first to Nth optical filters. Non-filtered light-sensitive cell groups that are not arranged to face each other, or transmittance averaged light in which optical filters having transmittances between the peak and under peak in the first to Nth optical filters are arranged to face each other A sensing cell group, and the signal processing circuit includes N types of photoelectric conversion signals from the first to Nth photosensitive cell groups and a photoelectric conversion signal from the non-filtered photosensitive cell group or the transmittance. average Using the photoelectric conversion signal from the photosensitive cell group, N types of periodic functions having the variable Z as the variable Z, and the change value of the variable Z with respect to the minute change of the wavelength λ, spectral characteristics of the captured image are obtained. calculate.

  According to the multispectral camera of one embodiment of the present invention, a mechanical mechanism that replaces the optical filter and scans the image sensor can be eliminated, and thus the movement of the subject can be dealt with. In addition, since the imaging element uses an optical filter having a plurality of peaks in transmittance in a predetermined wavelength band, the sensitivity can be made higher than when a conventional color filter such as RGB is used. In addition, since the signal from each photosensitive cell contains a large amount of color information, the reflected light spectrum of the subject can be approximated by combining these signals.

It is a typical spectral characteristic schematic diagram of the color filter in the embodiment of the present invention. (A) is a top view of the pixel basic unit of the image pick-up element in Embodiment 1 of this invention, (b) is an AA 'sectional view, (c) is a BB' sectional view, (d). Is a cross-sectional view taken along the line CC ′. (A) is the spectral characteristic figure of the simulated color filter 1a-1d in Embodiment 1 of this invention, (b) is the spectral characteristic figure of the simulated color filter 1e-1h. (A) is a characteristic diagram showing original data obtained by extracting only the variation component of the transmittance of the color filter 1a according to Embodiment 1 of the present invention, data obtained by variable conversion with respect to the variation component, and cosine function (cos) data. ) Is a similar characteristic diagram regarding the color filter 1b, (c) is a similar characteristic diagram regarding the color filter 1c, and (d) is a similar characteristic diagram regarding the color filter 1d. (A) is a characteristic diagram showing original data obtained by extracting only the variation component of the transmittance of the color filter 1e according to the first embodiment of the present invention, data obtained by variable conversion of the variation component, and cosine function (cos) data. ) Is a similar characteristic diagram regarding the color filter 1f, (c) is a similar characteristic diagram regarding the color filter 1g, and (d) is a similar characteristic diagram regarding the color filter 1h. It is a block diagram of the multispectral camera in Embodiment 1 of this invention. It is the figure which showed typically a mode that it images on an image pick-up element through the imaging lens and broadband optical filter in Embodiment 1 of this invention. (A) is a reflection spectral characteristic figure of Macbeth color checker No13-No15 made from X-Rite used as a photographic subject in Embodiment 1 of the present invention, and (b) is a reflection spectral characteristic figure of Macbeth color checker No16-No18. It is. FIG. 3 is a spectral characteristic diagram in which the spectral characteristics of an optical system including a light source and a lens, excluding the color filter according to Embodiment 1 of the present invention, and the spectral sensitivity of an image sensor are combined. It is a flowchart of imaging and signal processing simulation in Embodiment 1 of the present invention. (A) is the figure which showed the value of pixel E (0)-E (8) of the image pick-up element in Embodiment 1 of this invention, (b) is the calculated Fourier coefficient a (0)-a (8). It is the figure which showed the value of. (A) is a spectral characteristic diagram of a subject calculated in the first embodiment of the present invention, and (b) is a spectral characteristic diagram of a color checker that is a subject. It is a spectral characteristic figure of the color filter hold | maintained in the camera in Embodiment 2 of this invention. (A) is a top view of the image pick-up element containing the compound-eye optical system in Embodiment 3 of this invention, (b) is AA 'sectional drawing, (c) is BB' sectional drawing, (d) is CC 'sectional drawing. It is. It is a block diagram of the multispectral camera in Embodiment 3 of this invention. It is the figure which showed typically a mode that it images on an image pick-up through the broadband optical filter and nine imaging lens in Embodiment 3 of this invention. It is a figure which shows the modification of Embodiment 3 of this invention.

(Embodiment 1)
The multispectral camera according to the first embodiment of the present invention has an image pickup device having N + 1 (N is an integer of 2 or more) photosensitive cells as a basic unit of pixel, and the basic unit (hereinafter referred to as “pixel unit”). Each of the N light-sensitive cells in the “opposite” is opposed to one optical filter (hereinafter also referred to as “color filter”). That is, N optical filters are respectively arranged to face N photosensitive cells in each basic unit. Each of the N optical filters has one or more transmittance peaks or under peaks in a predetermined wavelength band (for example, 380 nm to 760 nm). Further, the N optical filters differ in the total number of peaks obtained by combining the number of transmittance peaks and the number of under peaks. That is, the basic unit of the pixel is composed of N pixels in which a color filter is arranged on the photosensitive cell and one pixel in which no color filter is arranged.

  In this specification, “peak” refers to one continuous partial band whose transmittance exceeds the average value in the predetermined wavelength band (in which the transmittance is equal to or lower than the average value). No) means that the transmittance is maximized. On the other hand, “under peak” means that the transmittance is minimized within one continuous partial band in which the transmittance is below the above average value (in which the transmittance does not exceed the average value). To do.

  All the color filters have, for example, a peak at the lower limit wavelength of the predetermined wavelength band and a peak or an under peak at the upper limit wavelength of the predetermined wavelength band. For example, among the N color filters, the first color filter may be an optical filter having a total peak number of two having one peak and one under peak between predetermined wavelength bands. The second color filter may be an optical filter having a total peak number of 3 having two peaks and one under peak between predetermined wavelength bands. The third color filter may be an optical filter having a total peak number of 4 having two peaks and two under peaks between predetermined wavelength bands. Similarly, the i-th (i is an integer greater than or equal to 1 and less than or equal to N) optical filter has a total peak number of i + 1. Thus, the N color filters have a feature that the total number of peaks increases by one as the number of codes increases. The light transmission characteristics of each color filter have variability with respect to the wavelength.

  For each pixel that receives light through these color filters, in the following, a pixel in which the first color filter is arranged is referred to as a first pixel, and a pixel in which the second color filter is arranged is referred to as a second pixel. The code number of the color filter is matched with the code number of the pixel. In this specification, a pixel may be referred to as a photosensitive cell.

  From the above pixel configuration, when the pixel signal is analyzed for each basic unit of the pixel, the pixel not provided with the color filter generates a signal obtained by photoelectrically converting all the light in the predetermined wavelength band. The first pixel generates a signal obtained by photoelectrically converting light including a particularly large light component in one specific (relatively narrow) wavelength band among all the light components in a predetermined wavelength band, and the second pixel , A signal obtained by photoelectrically converting light including particularly a large amount of light components in two specific (relatively narrow) wavelength bands out of all the light components in a predetermined wavelength band is generated. In this way, the i-th pixel (i is an integer of 1 or more and N or less) is light that contains a particularly large amount of light components in i specific (relatively narrow) wavelength bands out of all light components in a predetermined wavelength band. To generate a photoelectrically converted signal.

  When considering a function with the wavelength as the independent variable and the subject's light reflectance as the dependent variable, the signal of the pixel without the color filter in the predetermined wavelength band is considered to have a direct current component of the subject's light reflectance. . On the other hand, the signals of the first to Nth pixels are considered to have a part of the DC component and N types of AC components in the light reflectance distribution of the subject. Therefore, using these signals, it is possible to approximate the light reflectance of the subject in the form of a finite Fourier series. This approximation method is the approach of this embodiment.

Here, the basic principle of this embodiment will be described. Assuming that the function of the wavelength representing the reflected light energy from a certain point of the subject exists in a predetermined wavelength band λ1 to λ2 (where λ2−λ1 = W), the function is expressed as a shift wavelength X (0 ≦ 0) from the wavelength λ1. X (W) is used to express F (X). F (X) is a function that is defined only in the range of 0 ≦ X ≦ W. However, in the following explanation, it is assumed that F (X) is an even function, and that the expansion is performed with a Fourier cosine series. Think. Then, F (X) can be approximated by a finite Fourier cosine series expressed by Equation 1. In the following description, the shift wavelength X may be simply expressed as the wavelength X.

However, a (i) in Formula 1 is represented by the following Formula 2, and i is an integer from 0 to a maximum natural number M set in advance. Further, the integration range in Equation 2 is X = 0 to W.

  If a (i) shown in Expression 2 can be generated from the pixel signal of the image sensor, the reflection spectral characteristic of the subject can be approximately calculated. However, for this purpose, each color filter of the image sensor needs to have spectral characteristics that change in a cosine manner with respect to a change in wavelength. Such a color filter is a rather special optical filter. This embodiment also relieves the peculiarity of such a color filter, and the spectral characteristics of the color filter to be used need not necessarily change in a cosine manner with respect to a change in wavelength.

  For example, it is assumed that one of the color filters arranged on the pixel of the image sensor has a spectral characteristic indicated by a solid line in FIG. 1, and the characteristic is represented by G (X). G (X) has an average transmission ratio Ka (hereinafter also referred to as transmittance) when the maximum value of the light transmittance is 1, and the transmittance varies depending on the wavelength. It is assumed that a component (also referred to as an AC component) is represented by (PP / 2) × AC (X) using a difference PP between the maximum value and the minimum value. That is, G (X) = Ka + (PP / 2) AC (X). In addition, in a pixel in which no color filter is arranged, the amount of received light is slightly reduced due to light reflection on the surface of the image sensor, and the transmission ratio is Kd. For reference, the characteristic is shown by a broken line in FIG.

Assuming that light from the subject is 100% photoelectrically converted, the signal Sd of the pixel without the color filter is expressed by the following Expression 3, and the signal Sa of the pixel with the color filter is expressed by the Expression 4. .

Expression 5 is derived from Expression 3 and Expression 4. It can be seen that the intensity of the light when the reflected light from the subject is attenuated by the AC component of the color filter is obtained by calculating the pixel signal.

  If AC (X) of any color filter is a cosine function, the Fourier coefficient shown in Equation 2 can be obtained, and the reflection spectral characteristic F (X) of the subject can be calculated from Equation 1 using the result. However, when the spectral characteristics of each color filter have variability as shown in FIG. 1 and have an AC component that is not a cosine function, Equations 2 and 1 cannot be used.

と近似できる。Ｚ'＝ｄＺ／ｄＸとすると、式５は式６で表される。

The present embodiment solves this problem. If a certain condition is satisfied, the reflection spectral characteristic of the subject can be calculated even if a change in the spectral characteristic of the color filter is not represented by a cosine function. The certain condition is that, in any color filter, the AC component AC (X) of the spectral characteristic is close to a cosine function with respect to the change of the variable Z related to the wavelength X. If this condition is satisfied, the left side of Equation 5 is

Can be approximated. When Z ′ = dZ / dX, Expression 5 is expressed by Expression 6.

  Equation 6 means that the Fourier coefficient of (F (Z) / Z ′) can be obtained by calculation using the pixel signals Sa and Sd. As a result, a result obtained by expanding (F (Z) / Z ′) to a finite Fourier series is obtained, by multiplying the expanded result by Z ′, and further converting F (Z) to F (X). The reflection spectral characteristic of the subject can be obtained.

  The above is the basic principle of this embodiment. Hereinafter, a specific example of this embodiment will be described based on the basic principle.

  First, the image sensor of the multispectral camera in this embodiment will be described. In the present embodiment, an imaging device having a plurality of pixel units arranged two-dimensionally on the imaging surface is used. The pixel unit is a basic unit of pixels in the image sensor, and includes a plurality of photosensitive cells.

  FIG. 2 shows a basic unit of pixels of the image sensor. FIG. 2A is a plan view showing a basic unit of a pixel. In this embodiment, the basic configuration is 9 pixels, 3 pixels in the horizontal direction and 3 pixels in the vertical direction. A color filter is not disposed on the central pixel, and a color filter is disposed on the other pixels. 2B, 2C, and 2D are a cross-sectional view taken along line A-A ', a cross-sectional view taken along line B-B', and a cross-sectional view taken along line C-C 'in FIG. As shown in FIG. 2, the multispectral camera of the present embodiment has nine photosensitive cells 2a, 2b, 2c, 2d, 2e, 2f, 2g, 2h, 2z arranged in 3 rows and 3 columns. And eight color filters 1a, 1b, 1c, 1d, 1e, 1f, 1g, 1h. The color filters 1a, 1b, 1c, 1d, 1e, 1f, 1g, and 1h face the photosensitive cells 2a, 2b, 2c, 2d, 2e, 2f, 2g, and 2h, respectively. In this specification, such a facing relationship may be expressed as “corresponding”. The photosensitive cell 2z is located at the center of the pixel unit, and there is no color filter thereon. Each photosensitive cell has a light receiving element such as a photodiode that outputs a photoelectric conversion signal (also referred to as a pixel signal) according to the amount of light received. These photosensitive cells are formed on the silicon substrate 3 and face the plurality of color filters through the silicon oxide film 31.

  Image formation on the image sensor is photoelectrically converted through a color filter or directly in a light-sensitive cell. Although the photoelectric conversion signal from each photosensitive cell is omitted in FIG. 2, it is read out from the image sensor as an electrical signal through the wiring layer.

  Here, the color filters 1a to 1h will be described. The color filters 1a to 1h are one-layer titanium-based oxide films, but have different film thicknesses. In the present embodiment, the measurement wavelength band of the subject is set to 380 nm to 760 nm, and the film thicknesses of the color filters 1a to 1h are designed. The design condition is that light vertically enters the color filters 1a to 1h and is photoelectrically converted by the photosensitive cells 2a to 2h and 2z through the silicon oxide film 31, and the upper portions of the color filters 1a to 1h have a refractive index of 1. 0.0 air layer, each color filter is formed of a titanium-based oxide film having a refractive index of 2.5, and the lower part of the color filters 1a to 1h is a silicon oxide film 3 having a refractive index of 1.47. Under such design conditions, the thicknesses of the color filters 1a, 1b, 1c, 1d, 1e, 1f, 1g, and 1h are 76 nm, 152 nm, 228 nm, 304 nm, 380 nm, 456 nm, 532 nm, and 608 nm, respectively. The above film thickness is an integral multiple of the basic film thickness of 76 nm. As a result, in the wavelength band of 380 nm to 760 nm, the transmission characteristics of all the color filters have a peak at 380 nm, and have a peak or under peak at 760 nm.

  FIG. 3A shows the simulation results of the spectral characteristics of the color filters 1a to 1d, and FIG. 3B shows the simulation results of the spectral characteristics of the color filters 1e to 1h. These color filters have variability with respect to wavelength in the transmission ratio when the maximum value of the light transmittance is 1. The total number of peaks and under peaks in the spectral characteristics of each color filter is 2 in the wavelength band 380 nm to 760 nm, 2 for the color filter 1a, 3 for the color filter 1b, 4 for the color filter 1c, and 1 for the color filter 1d. There are five, six color filters 1e, seven color filters 1f, eight color filters 1g, and nine color filters 1h.

For the transmission ratio of each color filter, they repeatedly vary with increasing wavelength, but their period is not constant. However, it can be a constant period for another variable related to wavelength. In this embodiment, the wavelength is λ, and the transmittance of each color filter is Y. For example, Z expressed by a quadratic function of λ shown in Equations 7 and 8 (Z is a shift wavelength from the wavelength of 380 nm) is an independent variable. Then, the fluctuation component of the transmittance Y can be approximated by a cosine function.
(Formula 7) (Z + 380) = λ− (65 / 19.6) × (λ−520) ² +65 [when 380 ≦ λ ≦ 520 nm]
(Formula 8) (Z + 380) = λ− (65 / 57.6) × (λ−520) ² +65 [in the case of 520 ≦ λ ≦ 760 nm]

  4 and 5, the original data obtained by extracting only the variation component of the transmittance of each color filter, the data obtained by variable conversion of the variation component using Equations 7 and 8, and the cosine function (cos) data are shown. Show. 4A shows the data of the color filter 1a, FIG. 4B shows the data of the color filter 1b, FIG. 4C shows the data of the color filter 1c, and FIG. 4D shows the data of the color filter 1d. 5A shows the data of the color filter 1e, FIG. 5B shows the data of the color filter 1f, FIG. 5C shows the data of the color filter 1g, and FIG. 5D shows the data of the color filter 1h. Represents the data. In any of the figures, the solid line indicates the original data of the transmittance variation component of the color filter, the broken line indicates the variable-converted data regarding the variation component, and the one-dot chain line indicates the cosine function (cos) data. As can be seen from FIGS. 4 and 5, the variation component of the transmittance of each color filter is represented by a variable Z related to the wavelength λ, so that it can be seen that the distribution is almost similar to a distribution represented by a cosine function. As described above, in the present specification, changing approximately like a periodic function is expressed as “periodically changing”. A periodic change in this specification does not necessarily mean a periodic change in a strict sense.

  Next, pixel signals of the image sensor will be described. The pixels in which the color filters 1a to 1h are arranged in this order are referred to as first to eighth pixels, and their pixel signals are represented by E (1) to E (8). A pixel in which no color filter is arranged is referred to as a ninth pixel, and the pixel signal is represented by E (0).

In the following description, the symbols shown in FIG. 1 are used as they are for the characteristics of each color filter and the spectral characteristics of the pixels where no color filter is arranged. In the wavelength band 380 nm to 760 nm, the pixel signals E (0) and E (1) to E (8) are expressed using the shift wavelength X from the lower limit wavelength 380 nm and the function F (X) indicating the reflection spectral characteristics of the subject. And represented by the following formulas 9 and 10.

  However, regarding the input variable of AC (i, X), i is a numerical value (1 to 8) numbered in the order of the color filters 1a to 1h, and X is the above wavelength. That is, AC (1, X) to AC (8, X) are functions of characteristics indicated by solid lines in FIGS. 4 (a) to 4 (d) and FIGS. 5 (a) to 5 (d) in order. is there.

Equation 11 is obtained from Equation 9 and Equation 10.

As AC (i, X) can be approximated by a cosine function as shown in FIGS. 4 and 5, Equation 11 can be expressed by Equation 12.

When Z ′ = dZ / dX, Expression 12 is expressed by Expression 13.

For the Fourier coefficient of F (Z) / Z ′, a (0) is (2 / W) ∫ (F (Z) / Z ′) dZ, which is (2 / W) ∫F (X) dX The same. For this reason, a (0) is expressed by Expression 14 using Expression 9. On the other hand, a (i) where i ≧ 1 is expressed by Expression 15 using Expression 13 based on Expression 2.

From the Fourier coefficients obtained by Expressions 14 and 15, F (Z) / Z ′ can be approximately expanded by the Fourier series shown in Expression 16. By multiplying the result by the value of Z ′, F (Z (X)), that is, F (X) is obtained.

  As described above, the Fourier coefficient is calculated using the pixel signal of the image sensor, and the reflection spectral characteristic of the subject can be approximately calculated by multiplying the Fourier series expansion of Equation 1 and Z ′ using them.

  Next, the configuration of the multispectral camera and signal processing in the camera according to the present embodiment will be described. FIG. 6 is a configuration diagram of the multispectral camera in the present embodiment. The multispectral camera includes an imaging lens 11, a broadband optical filter 12, an image sensor 13, a signal generation / reception unit 14, an image processing unit 15, an image memory 16, and a signal output unit 17. Yes. The imaging lens 11 is depicted as a single lens in FIG. 6, but may be a combination of a plurality of lenses. In the present embodiment, the broadband optical filter 12 is designed to transmit only light having a wavelength band of 380 nm to 760 nm. The image sensor 13 is a CMOS type image sensor in the present embodiment, but may be another type of image sensor such as a CCD. The signal generation / reception unit 14 receives an image signal from the image sensor 13 and sends a signal for driving the image sensor 13 to the image sensor 13. The signal generation / reception unit 14 may be composed of an LSI such as a CMOS driver, for example. The image processing unit 15 transmits the image signal from the signal generation / reception unit 14 to the image memory 16 and reads and processes the image signal from the image memory 16. The image processing unit 15 can be realized, for example, by a combination of a signal processing circuit such as a known digital signal processor (DSP) and software that executes image processing in the present embodiment. Alternatively, the image processing unit 15 may be configured by dedicated hardware. The signal processing in the present embodiment is executed by a signal processing circuit in the image processing unit 15. The image memory 16 can be constituted by a known semiconductor memory such as DRAM or SRAM. The signal output unit 17 outputs the signal from the image processing unit 15 to the outside.

  Of the light from the subject, only the light in the wavelength band of 380 nm to 760 nm is transmitted by the imaging lens 11 and the broadband optical filter 12 and imaged on the imaging unit (imaging surface) of the imaging device. FIG. 7 is a diagram schematically showing a state in which light from a subject passes through the imaging lens 11 and the broadband optical filter 12 and is imaged on the imaging unit (imaging surface) 13 a of the imaging device 13. The formed image is photoelectrically converted by the image sensor 13 to become an electric signal, which is recorded in the image memory 16 via the signal generation / reception unit 14 and the image processing unit 15.

  The image processing unit 15 reads the image information recorded in the image memory 16 for each basic unit of the pixel and performs signal processing. As an example, imaging and signal processing simulation for calculating the spectral characteristics of a subject when imaging color samples of Macbeth Color Checker No. 13 to No. 18 manufactured by X-Rite will be described below. The color samples are blue (B), green (G), red (R), yellow (Ye), magenta (Mg), and cyan (Cy) in the order of No13 to No18. Their reflection spectral characteristics are shown in FIG. FIG. 4A shows the reflection spectral characteristics of No. 13 to No. 15 and is represented by a solid line, a broken line, and a one-dot chain line in order of numbers. FIG. 4B shows the reflection spectral characteristics of No. 16 to No. 18 and is represented by a solid line, a broken line, and an alternate long and short dash line in numerical order.

  The imaging conditions are as follows. A halogen lamp with a color temperature of 5100K was used as the light source. FIG. 9 shows the spectral characteristics of the optical system including the light source and lens, excluding the color filter, and the spectral sensitivity of the image sensor. This characteristic is called an imaging optical spectral characteristic and is represented by O (X). However, X is a shift wavelength from 380 nm. In this embodiment, since the wavelength band of 380 nm to 760 nm is targeted, the possible value of X is 0 to 380 nm. Further, with respect to the coefficients relating to the image sensor including the color filter, Ka = 0.79, PP = 0.347, and Kd were 0.96 based on FIG.

  Hereinafter, the flow of simulation will be described with reference to FIG. However, as a pre-process, Equations 7 and 8 are differentiated by the wavelength λ, and Z ′ (= dZ / dX) is calculated as λ = 380 + X.

  First, as a first step, assuming that each of the color samples No. 13 to No. 18 has been imaged, a discrete integration operation of their spectral characteristics, imaging optical spectral characteristics O (X), and spectral transmission characteristics of each color filter is performed, and each color sample is obtained. Signal values E (0) to E (8) relative to the sample are calculated (S1). In actual calculation, discrete integration is performed using numerical data every 10 nm, and nine signal values are calculated for each color sample. Therefore, a total of 6 × 9 = 54 signal values are calculated.

  Next, as a second step, the calculated signal values E (0) to E (8) are multiplied by the correction coefficient Kg (S2). As can be seen from FIG. 4 and FIG. 5, in the relationship between the transmittance and wavelength of each color filter, the waveform (dashed line in the figure) when the wavelength λ is converted into the variable Z (the broken line in the figure) is an ideal cosine function waveform (FIG. This is performed in order to correct an error caused by being lower than the middle one-dot chain line). Since the waveform (dashed line in the figure) when the wavelength λ is converted to the variable Z is lower than the ideal cosine function waveform (dashed line in the figure), the signal values E (0) to E (8) are It is considered to be smaller than the ideal signal value. Therefore, each signal is multiplied by a correction coefficient Kg to slightly amplify the signal. In this embodiment, the correction coefficient Kg is 1.025.

  As a third step, the result of multiplying the correction coefficient Kg is newly set as signal values E (0) to E (8), and the Fourier coefficients a (0) to a (8) of each color sample using Expressions 14 and 15. ) Is calculated (S3). FIG. 11A shows signal values E (0) to E (8) after being multiplied by the correction coefficient Kg, and FIG. 11B shows the calculated Fourier coefficients a (0) to a (8). However, these numerical values are relative values in calculation.

  As a fourth step, F (Z) / Z ′ is expanded into a Fourier series using the calculated Fourier coefficients a (0) to a (8) (S4). As a fifth step, F (Z (X)), that is, F (X) is calculated by multiplying the expanded Fourier series by Z '(S5). The calculated F (X) is the spectrum of the captured image, which includes the imaging optical spectral characteristics described above. Therefore, in the sixth step, F (X) is further divided by O (X) to calculate the spectral characteristics of the subject (S6).

  FIG. 12A shows the calculated spectral characteristics of the subject, and FIG. 12B shows the actual spectral characteristics of the color checker that is the subject. In FIGS. 12A and 12B, the thick solid line is blue (calculation result: Cal_No13, actual characteristic: No13), the thick broken line is green (calculation result: Cal_No14, actual characteristic: No14), and the thick one-dot chain line Is red (calculation result: Cal_No15, actual characteristic: No15), thin solid line is yellow (calculation result: Cal_No16, actual characteristic: No16), thin broken line is magenta (calculation result: Cal_No17, actual characteristic: No17), thin A one-dot chain line represents a spectral characteristic of a color checker of cyan (calculation result: Cal_No18, actual characteristic: No18). In FIG. 12, although the wavelength range is 400 nm to 700 nm, it can be seen that the calculated spectral characteristics of the subject are substantially the same as the spectral characteristics of the color checker.

  Finally, from the calculated spectral characteristics of the subject, spectral characteristic values at pre-specified wavelengths are extracted, and a specific wavelength image using these data as pixel signals and the average value of the pixel signals for each basic unit of the pixel are calculated. Then, a black and white image using these as pixel signals is output to the outside from the signal output unit 17 (S7). In this way, the specific wavelength image and the monochrome image are output from the image processing unit 15 via the signal output unit 17. All images within the set measurement wavelength band can be obtained by changing the designated wavelength. In addition, for the output black and white image, a number of color filters having a plurality of spectral transmittance peaks are arranged in the pixels of the image sensor 13, so that the conventional monochrome image has only one spectral transmittance peak. The light utilization rate is higher than when RGB filters are used, and therefore the brightness of the image is high.

  As described above, in the multispectral camera of the present embodiment, N = 8, and there is at least one transmittance peak or under peak in a predetermined wavelength band, and the number of peaks and the number of under peaks are set as follows. First to eighth optical filters having different total peak numbers are used. In the first to eighth filters, in the relationship between the wavelength λ and the respective transmittance Y in the predetermined wavelength band, the transmittance Y is periodic with respect to the change of the variable Z associated with the wavelength λ. Changes. Further, any of the first photosensitive cell to the eighth photosensitive cell in which each of the first to eighth filters is arranged correspondingly (opposed) and the first to eighth optical filters are arranged. An image sensor is used in which the unfiltered light-sensitive cell that is not used is a pixel unit. Nine photoelectric conversion signals E (0) to E (8) from the non-filter photosensitive cell and the first photosensitive cell to the eighth photosensitive cell, and eight types with the independent variable as the variable Z And a change value (Z ′ = dZ / dX) of the variable Z with respect to a minute change of the wavelength λ (equivalent to a minute change of the shift wavelength X), and a periodic function AC (1, X) to AC (8, X) of Can be used to calculate the spectral characteristics of the captured image (that is, the intensity distribution for each wavelength). Further, the reflection spectral characteristic of the subject can be calculated by dividing the calculated spectral characteristic by the spectral characteristic of the imaging optical system. In addition, since an optical filter having a plurality of transmittance peaks is used, the captured image obtained from the camera has a high brightness.

  In the present embodiment, among the first to Nth optical filters, the i-th (i is an integer not less than 1 and not more than N) optical filter has a total peak number of i + 1. The N types of periodic functions used in the calculation are N cosine functions cos (iZ) having a frequency that is an integral multiple of 1 to N times a predetermined frequency. In the above description, the expression cosine function cos (iπZ / W) is used. However, since Z may be a variable related to the wavelength λ, the above-described πZ / W may be expressed as Z again. it can. When rewritten in this way, the aforementioned cosine function is expressed as cos (iZ).

  The signal processing circuit in the image processing unit 15 sets E (0) as the photoelectric conversion signal of the non-filter photosensitive cell and E (i) as the photoelectric conversion signal of the i-th photosensitive cell. The result a (0) obtained by multiplying the constant k1 (2 / WKd in the above example) and the result obtained by multiplying E (i) by the second constant k2 (2 in the above example) The cosine function cos (iZ) is subtracted from the result a (i) obtained by subtracting the result (2 (Ka / Kd) in the above example) obtained by multiplying the third constant and further dividing by the fourth constant (PP). A (0) and a (1) cos (Z) -a (N) cos (NZ) are added to each other using the multiplied a (i) cos (iZ). The reflection spectral characteristic of the subject is calculated by multiplying the change value of the variable Z with respect to the minute change of.

  As shown in step S2 of FIG. 10, the signal processing circuit corrects the signal value by multiplying the photoelectric conversion signal from the first to Nth photosensitive cells by a fifth constant (Kg). Also good.

  In this embodiment, the basic unit of pixels is 9 × 3 × 3 pixels, but is not limited to this. For example, there is no problem even if an integer pixel of U × V (U and V are natural numbers different from each other) is used as a basic unit. Further, if only the spectral characteristic waveform (AC component) of the subject is to be measured, a pixel in which a color filter is not arranged (filter-free photosensitive cell) related to the calculation of the DC component of the spectral characteristic in the basic unit of the pixel. May not be included. In that case, the number of photosensitive cells included in the pixel unit and the number of optical filters opposed to them coincide. Further, in place of the non-filter photosensitive cell, a transmittance averaged photosensitive cell in which optical filters having transmittances between the peak and under peak in the first to Nth optical filters are arranged to face each other is included. You can leave. In a high refractive index material, the refractive index varies greatly depending on the wavelength. In such a material, the average transmittance of the peak and the under peak is increased on the long wavelength side. In order to remove the influence, a filter having an average transmittance of all peaks and all under peaks may be provided. Further, the color filter is not limited to a single layer film, and may be composed of a plurality of layers. There are optical filter groups having different total number of peaks in the transmittance, and in the relationship between the wavelength λ and the respective transmittance Y, all the optical filters have a cycle of the transmittance Y with respect to the change of the variable Z related to the wavelength λ. Therefore, the configuration of the color filter is arbitrary. In addition, regarding the relationship between the variable Z and the wavelength λ, in the present embodiment, the approximate expression is expressed by Expression 7 and Expression 8, but the present invention is not limited to this, and another approximate expression using a spline function or the like may be used. There is no. In addition, regarding the structure of the image sensor, a micro lens is not disposed in each pixel in the present embodiment, but a micro lens may be disposed in order to improve the sensitivity of the image sensor. Further, regarding the signal processing in the camera, the signal is processed for each basic unit of the pixel, but the pixel signal of the adjacent basic unit may also be used to prevent false color. As described above, the signal processing circuit may perform signal processing for each pixel unit and generate interpolation information using a photoelectric conversion signal from an adjacent pixel unit. Furthermore, in the present embodiment, the measurement wavelength band is designed to be 380 nm to 760 nm. However, the measurement wavelength band is not limited to this, and a wider wavelength band may be used. Also, in the above description, the color filter is not implicitly colored and the refractive index is constant regardless of the wavelength, but in actuality, it is slightly colored and has a peak on the short wavelength side because the refractive index slightly changes depending on the wavelength. There is a problem that the difference in underpeak changes or the average transmittance decreases. Therefore, after the sixth step (S6) of the above processing, the reflection spectral characteristics of the subject may be further corrected using correction coefficients corresponding to these problems. Such a correction coefficient is set so as to cancel out the influence due to the attenuation of the light transmittance in the short wavelength region and the influence due to the difference in the amplitude of the transmittance depending on the wavelength.

(Embodiment 2)
Next, a second embodiment of the present invention will be described. The present embodiment has the same image sensor and camera configuration as the first embodiment, but has spectral characteristic data of three types of color filters for creating a color image in the image processing unit. The spectral characteristics of these color filters are shown in FIG. In the figure, the dotted line is the spectral characteristic of the R (red) filter, the broken line is the spectral characteristic of the G (green) filter, and the dashed line is the spectral characteristic of the B (blue) filter.

  In this embodiment, the processing up to the calculation of the spectral characteristics of the subject is the same as in the first embodiment. Thereafter, the spectral characteristics of the subject are multiplied by the spectral characteristics of the color filter, and the R signal, G signal, and B signal are obtained. It has processing to calculate. After the processing, a color image is created from the calculated RGB signals and output from the signal output unit to the outside. Normally, a color camera that outputs a color image requires an infrared cut filter, but this embodiment has a feature that an infrared cut filter is not required. Also, the transmittance of the color filter usually cannot take a negative value, but in the present embodiment, the transmittance of the color filter has a partial negative characteristic. A preferable color image can be produced. However, if the result of calculation as an RGB signal is a negative signal, a process for forcibly setting the signal value to 0 is performed.

  As described above, the multispectral camera according to the present embodiment has spectral characteristic data of three types of color filters inside, and a color signal can be generated by multiplying the calculated spectral characteristics of the subject. . That is, the signal processing circuit in the present embodiment uses the data indicating the spectral characteristics of the plurality of color filters to perform an integral operation on the calculated function indicating the spectral characteristics of the captured image and the function indicating the spectral characteristics of the color filter. Thus, a color signal is calculated. There is an advantage that a color image can be generated without using an infrared cut filter. Also, a color filter having negative spectral characteristics can be used for calculation. For this reason, there is an effect that it is possible to provide a color image with an unprecedented color reproduction range.

  In the present embodiment, the spectral characteristic data of three types of color filters are held in the camera. However, the present invention is not limited to this, and spectral characteristic data of four or more types of color filters may be held. The spectral characteristic data of each color filter is stored in a recording medium such as a memory in the camera, and the signal processing circuit can read the data and perform the above processing.

(Embodiment 3)
Subsequently, a third embodiment will be described. The multispectral camera according to the present embodiment is an embodiment in that the imaging surface of the imaging device is divided into a plurality of imaging regions, and a plurality of optical systems (for example, lenses) are arranged facing the imaging regions. Different from 1 and 2. Each imaging region includes a large number of photosensitive cells (referred to as photosensitive cell groups). Assuming that the number of imaging areas is N + 1, N optical filters are respectively arranged between N imaging areas of the imaging areas and N optical systems facing the imaging areas. These N optical filters have the same characteristics as the N optical filters (color filters) described in the first and second embodiments.

  The multispectral camera in the present embodiment photographs one subject using a plurality of optical systems. An imaging device including a compound-eye optical system in which N + 1 optical systems are arranged and N + 1 photosensitive cell groups (the above-described imaging regions) on the premise that substantially the same captured image can be obtained through each optical system. And at least N optical filters. These optical filters are respectively arranged corresponding to a plurality of optical systems included in the compound-eye optical system. Each optical filter has one or more transmittance peaks or under peaks in a predetermined wavelength band (for example, 380 nm to 760 nm), similarly to the optical filter in the first embodiment. The first to Nth optical filters (color filters) differ in the total number of peaks, which is the sum of the number of peaks and the number of underpeaks. That is, when the subject is imaged, N images captured through N types of color filters and one image captured without passing through the color filters are obtained. If attenuation by the color filter is not considered, they are all assumed to be the same image. However, the transmittance of the N types of color filters may attenuate as a whole as the wavelength becomes shorter. In such a case, an optical filter having an attenuation characteristic similar to the attenuation characteristic (hereinafter referred to as an averaging filter) may be arranged to face the optical system in which no color filter is arranged. In that case, an image through an averaging filter is also obtained. It can be said that the averaging filter has a transmittance between the peak and the under peak in the first to Nth color filters. The photosensitive cell group that receives the light transmitted through the averaging filter is referred to as “transmittance averaged photosensitive cell group”. On the other hand, when the averaging filter is not disposed, a photosensitive cell group that receives incident light without passing through any filter is referred to as an “unfiltered photosensitive cell group”.

  In the following description, the expression “Nth pixel” is a general term for the Nth pixel group, and may refer to a pixel at the same position in the image. In this case, the first to Nth pixels are pixels at the same position in each image. Here, the pixels at the same position in the image represent the first to Nth pixels and the pixels at the same position in the pixel group in which no color filter is arranged.

  Details of this embodiment will be described below.

  First, the image sensor of the multispectral camera in this embodiment will be described. In this embodiment, a two-dimensional image sensor having a compound eye optical system composed of nine optical systems is used. FIG. 14 shows the structure of the image sensor 13. FIG. 14A is a plan view of the image sensor 13 including a compound eye optical system. The image pickup device of this embodiment has nine imaging lenses 20 in total, three in the horizontal direction and three in the vertical direction. A color filter is disposed directly below the eight imaging lenses 20 among them. FIG. 14B shows a cross section taken along line A-A ′ in FIG. 14A, FIG. 2C shows a cross section taken along line B-B ′, and FIG. 14D shows a cross section taken along line C-C ′. As shown in FIG. 14, the image sensor 13 includes color filters 1a, 1b, 1c, 1d, 1e, 1f, 1g, and 1h, and photosensitive cell groups 2a, 2b, 2c, 2d, 2e, 2f, 2g, and 2h. 2z. On the surface of the image sensor 13, a plurality of photosensitive cell groups 2a to 2z are two-dimensionally arranged. As can be seen from the figure, there is no color filter on the photosensitive cell group 2z. The photosensitive cell group 2z directly photoelectrically converts an image formed by the imaging lens.

  The image formation on the image sensor 13 is photoelectrically converted through the color filters 1a to 1h or directly in the photosensitive cell. Although omitted in FIG. 14, the photoelectric conversion signal is read out as an electrical signal from the image sensor through the wiring layer. The subject is sufficiently away from the image sensor 13, and the images in the photosensitive cell groups 2a to 2z by the nine imaging lenses 20 are almost the same if the difference in the transmission characteristics of the optical filters 1a to 1h is ignored. . Furthermore, the photosensitive cell groups 2a to 2z have the same number of pixels and the same size. That is, the images read from these light sensitive cell groups are almost the same. If there is optical distortion in each optical system of the compound eye optical system, the image processing unit (signal processing circuit) may perform processing for correcting image distortion caused by the optical distortion. The color filters 1a to 1h in the present embodiment have the same characteristics as the color filters 1a to 1h in the first and second embodiments, although the sizes are different.

  Next, pixel signals of the image sensor 13 will be described. Here, pixels at the same place in the photosensitive cell groups 2a to 2h are referred to as first to eighth pixels in the order of the photosensitive cell groups 2a to 2h, and their pixel signals are represented by E (1) to E (8). In addition, the pixel at the same location in the photosensitive cell group 2z is referred to as a ninth pixel, and the pixel signal is represented by E (0). Then, the processing of Expression 9 to Expression 15 described in Embodiment 1 can be applied as it is, and the reflection spectral characteristics of the subject can be calculated approximately.

  Next, the configuration of the multispectral camera and signal processing in the camera according to the present embodiment will be described. FIG. 15 is a configuration diagram of the multispectral camera in the present embodiment. The multispectral camera of the present embodiment has the same configuration as that of the first and second embodiments except that nine optical filters 1 and nine imaging lenses 20 are arranged facing the image sensor 13. . For this reason, the description of the overlapping components is omitted.

  Of the light from the subject, only the light in the wavelength band of 380 nm to 760 nm is transmitted by the broadband optical filter 12, and is imaged on the imaging unit (imaging surface) of the imaging element 13 by the nine imaging lenses 20. FIG. 16 is a diagram schematically showing a state in which light from a subject passes through the broadband optical filter 12 and the nine imaging lenses 20 and is imaged on the imaging unit 13a of the imaging device 13. The imaging unit is virtually divided into nine imaging regions (2a to 2h and 2z) corresponding to the nine imaging lenses 20. The formed image is photoelectrically converted by the image sensor 13 to become an electric signal, which is recorded in the image memory 16 via the signal generation / reception unit 14 and the image processing unit 15.

  The image processing unit 15 reads the image information recorded in the image memory 16 for each of the nine virtual imaging areas of the pixels and performs signal processing. This signal processing is the same as the processing described in the first embodiment. Further, the processing described in the second embodiment may be applied.

  As described above, the multispectral camera according to the present embodiment has a compound-eye optical system including nine optical systems, one or more transmittance peaks or under peaks in a predetermined wavelength band, and the number of the peaks. The first to eighth optical filters having different total peaks combined with the number of under peaks are used. In the first to eighth filters, in the relationship between the wavelength λ and the respective transmittance Y in the predetermined wavelength band, the transmittance Y is periodic with respect to the monotonous increase of the variable Z associated with the wavelength λ. To change. Further, any of the first photosensitive cell group to the eighth photosensitive cell group in which the first to eighth filters are respectively arranged correspondingly and the first to eighth optical filters are also disposed. An image sensor having no filter-free photosensitive cell group is used. Nine photoelectric conversion signals from the non-filter photosensitive cell and the first to eighth photosensitive cells, N types of periodic functions having the independent variable Z as the variable Z, and the wavelength λ Using the change value of the variable Z with respect to a minute change, the spectral characteristic of the captured image can be calculated. Further, the reflection spectral characteristic of the subject can be calculated by dividing the calculated spectral characteristic by the spectral characteristic of the imaging optical system. Further, since an optical filter having a plurality of transmittance peaks is used, the captured image obtained from the camera has an advantage of high brightness.

  In the present embodiment, a compound-eye optical system using nine 3 × 3 lenses and eight color filters are used, but the present invention is not limited to this. For example, as shown in FIG. 17, a configuration in which a concave lens array 22 including a concave lens without a color filter and eight concave lenses each having the color filters 1a to 1h are arranged on the front surface of a normal monocular monochrome camera 21 may be used. . The concave lens array 22 may include a decentered concave lens. However, in such a configuration, when there is distortion due to the concave lens at a plurality of positions of the image or when the image position is shifted, it is necessary to correct those images. In such a configuration, a relatively large imaging lens 23 can be disposed between the concave lens array 22 and the image sensor 13 as shown in FIG. Such an imaging lens 23 focuses the light spread by each concave lens onto the pixel region on the image sensor corresponding to the concave lens. The reason why the concave lens is decentered is that an image is appropriately formed in the imaging region corresponding to the concave lens by optically shifting the imaging position. As another example, a configuration in which nine convex lens arrays including a convex lens without a color filter and eight convex lenses each having the color filters 1a to 1h are arranged on the front surface of a normal monocular black and white camera may be used. However, in that case, when there is distortion due to the convex lens at a plurality of positions of the image or when the image position is shifted, it is necessary to correct those images. Also in this case, the convex lens is decentered in order to form an image appropriately in the imaging region corresponding to the convex lens by optically shifting the imaging position.

  As described above, the present disclosure includes the multispectral cameras described in the following items.

[Item 1]
An imaging device in which a plurality of pixel units are two-dimensionally arranged on the imaging surface, each pixel unit including first to Nth (N is an integer of 2 or more) photosensitive cells, An image sensor that outputs a photoelectric conversion signal corresponding to the amount of received light; and
1st to Nth optical filters disposed respectively facing the first to Nth photosensitive cells, each having one or more transmittance peaks or underpeaks in a predetermined wavelength band And the first to Nth optical filters having different total peak numbers in which the number of peaks and the number of underpeaks are combined,
A signal processing circuit for processing photoelectric conversion signals output from the first to Nth photosensitive cells;
With
The transmittance Y of each of the first to Nth optical filters periodically changes with respect to the change of the variable Z related to the wavelength λ in the predetermined wavelength band,
The signal processing circuit includes N photoelectric conversion signals from the first to Nth photosensitive cells, N types of periodic functions having the independent variable as the variable Z, and the variable for the minute change in the wavelength λ. Calculate the spectral characteristics of the captured image using the change value of Z.
Multispectral camera.

[Item 2]
The pixel unit may be an unfiltered photosensitive cell that is not arranged corresponding to any of the first to Nth photosensitive cells and the first to Nth optical filters, or the first to Nth photosensitive cells. A transmittance averaged photosensitive cell in which an optical filter having a transmittance intermediate between the peak and the under peak in the optical filter is disposed;
The multispectral camera according to item 1, wherein the signal processing circuit calculates the spectral characteristic by further using a photoelectric conversion signal from the non-filtered photosensitive cell or the transmittance averaged photosensitive cell.

[Item 3]
A band transmission filter that transmits only light of a predetermined wavelength band;
The signal processing circuit captures spectral characteristics of a captured image calculated using the N photoelectric conversion signals acquired by imaging through the band-pass filter, excluding the first to Nth optical filters. Item 3. The multispectral camera according to item 1 or 2, wherein the spectral characteristic of the captured image excluding the influence of the imaging optical system is calculated by dividing by the spectral characteristic of the optical system.

[Item 4]
Of the first to N-th optical filters, the i-th (i is an integer greater than or equal to 1 and less than or equal to N) optical filter has a total peak number of i + 1, and the N types of periodic functions have a predetermined frequency. Item 4. The multispectral camera according to any one of Items 1 to 3, which is N cosine functions cos (iZ) having any one of integer multiples of 1 to N times.

[Item 5]
The signal processing circuit has E (0) as the photoelectric conversion signal of the non-filtered photosensitive cell or the transmittance averaged photosensitive cell, and E (i) as the photoelectric conversion signal of the i-th photosensitive cell.
The result a (0) obtained by multiplying E (0) by the first constant k1 and the result obtained by multiplying E (0) by the third constant are subtracted from the result obtained by multiplying E (i) by the second constant k2. And a (i) cos (iZ) obtained by multiplying the result a (i) obtained by dividing the fourth constant by the cosine function cos (iZ),
By multiplying the result of adding a (0) and all of a (1) cos (Z) to a (N) cos (NZ) by the change value of the variable Z with respect to the minute change of the wavelength λ, Item 3. The multispectral camera according to Item 2, wherein the spectral characteristic of the captured image is calculated.

[Item 6]
The signal processing circuit corrects photoelectric conversion signals from the first to Nth photosensitive cells by multiplying photoelectric conversion signals from the first to Nth photosensitive cells by a fifth constant. Item 6. The multispectral camera according to item 5.

[Item 7]
Item 2, 5, or 6, wherein in each pixel unit, the first to Nth photosensitive cells are arranged around the unfiltered photosensitive cell or the transmittance averaged photosensitive cell. A multispectral camera according to any one of the above.

[Item 8]
Item 8. The multispectral camera according to Item 7, wherein the signal processing circuit performs signal processing for each pixel unit and generates interpolation information using a photoelectric conversion signal from an adjacent pixel unit.

[Item 9]
Item 9. The multispectral camera according to Item 7 or 8, wherein the imaging device has a plurality of microlenses arranged for each pixel unit.

[Item 10]
The signal processing circuit uses data indicating spectral characteristics of a plurality of color filters to perform an integral operation on the calculated function indicating the spectral characteristics of the captured image and the function indicating the spectral characteristics of the color filter. The multispectral camera according to any one of items 1 to 9, which calculates a color signal.

[Item 11]
A compound eye optical system including N + 1 (N is an integer of 2 or more) optical systems;
1st to Nth optical filters having one or more transmittance peaks or underpeaks in a predetermined wavelength band, and having different total peaks combined with the number of peaks and the number of underpeaks;
An imaging device having first to Nth photosensitive cell groups opposed to the first to Nth optical filters, each of the photosensitive cells outputting a photoelectric conversion signal corresponding to the amount of received light;
A signal processing circuit for processing photoelectric conversion signals output from the first to Nth photosensitive cells;
With
The transmittance Y of each of the first to Nth optical filters periodically changes with respect to the change of the variable Z related to the wavelength λ in the predetermined wavelength band,
The first to Nth optical filters are respectively arranged to face N optical systems of the N + 1 optical systems in the compound eye optical system,
The image pickup device further includes an unfiltered photosensitive cell group in which none of the first to Nth optical filters are arranged to face each other, or an intermediate between a peak and an under peak in the first to Nth optical filters. A transmittance averaged photosensitive cell group disposed oppositely with an optical filter having a transmittance of
The signal processing circuit includes N types of photoelectric conversion signals from the first to Nth photosensitive cell groups, a photoelectric conversion signal from the non-filter photosensitive cell group, or the transmittance averaged photosensitive cell group. A spectral characteristic of the captured image is calculated using a photoelectric conversion signal of N, a periodic function of N types having an independent variable as the variable Z, and a change value of the variable Z with respect to a minute change of the wavelength λ.
Multispectral camera.

[Item 12]
A band transmission filter that transmits only light of a predetermined wavelength band;
The signal processing circuit has the first to Nth optical filters for the spectral characteristics of the captured image calculated using the N types of photoelectric conversion signals acquired by imaging through the band pass filter. Item 12. The multispectral camera according to Item 11, wherein when the transmittance peak or under peak attenuates or increases with respect to wavelength change, the spectral characteristic of the captured image in which the attenuation or increase is corrected is calculated.

[Item 13]
A band transmission filter that transmits only light of a predetermined wavelength band;
The signal processing circuit captures spectral characteristics of a captured image calculated using the N types of photoelectric conversion signals acquired by imaging through the band-pass filter, excluding the first to Nth optical filters. Item 13. The multispectral camera according to item 11 or 12, wherein the spectral characteristic of the captured image excluding the influence of the imaging optical system is calculated by dividing by the spectral characteristic of the optical system.

[Item 14]
Of the first to N-th optical filters, the i-th (i is an integer greater than or equal to 1 and less than or equal to N) optical filter has a total peak number of i + 1, and the N types of periodic functions have a predetermined frequency. Item 4. The multispectral camera according to any one of Items 1 to 3, which is N cosine functions cos (iZ) having any one of integer multiples of 1 to N times.

[Item 15]
The signal processing circuit has E (0) as the photoelectric conversion signal of the unfiltered photosensitive cell, and E (i) as the photoelectric conversion signal of the i-th photosensitive cell.
The result a (0) obtained by multiplying E (0) by the first constant k1 and the result obtained by multiplying E (0) by the third constant are subtracted from the result obtained by multiplying E (i) by the second constant k2. And a (i) cos (iZ) obtained by multiplying the result a (i) obtained by dividing the fourth constant by the cosine function cos (iZ),
By multiplying the result of adding a (0) and all of a (1) cos (Z) to a (N) cos (NZ) by the change value of the variable Z with respect to the minute change of the wavelength λ, Item 13. The multispectral camera according to item 11 or 12, which calculates the spectral characteristic of a captured image.

[Item 16]
The signal processing circuit corrects photoelectric conversion signals from the first to Nth photosensitive cells by multiplying photoelectric conversion signals from the first to Nth photosensitive cells by a fifth constant. Item 16. The multispectral camera according to item 15.

[Item 17]
The multispectral camera according to any one of items 11 to 16, wherein the signal processing circuit calculates a spectral characteristic of the captured image after correcting distortion of an image formed by each optical system in the compound eye optical system. .

[Item 18]
The multispectral camera according to any one of items 11 to 17, wherein the compound eye optical system includes a plurality of convex lenses.

[Item 19]
Item 19. The multispectral camera according to Item 18, wherein the plurality of convex lenses includes a decentered convex lens.

[Item 20]
18. The multispectral camera according to any one of items 11 to 17, wherein the compound eye optical system includes a plurality of concave lenses and one convex lens.

[Item 21]
Item 21. The multispectral camera according to Item 20, wherein the plurality of concave lenses includes a decentered concave lens.

[Item 22]
The signal processing circuit uses data indicating spectral characteristics of a plurality of color filters to perform an integral operation on the calculated function indicating the spectral characteristics of the captured image and the function indicating the spectral characteristics of the color filter. Item 22. The multispectral camera according to any one of Items 11 to 21, which calculates a color signal.

  The multispectral camera of the present invention is effective for all solid-state cameras mainly using a solid-state imaging device. For example, it can be used for consumer cameras such as digital still cameras and digital video cameras, and industrial surveillance cameras.

1a, 1b, 1c, 1d, 1e, 1f, 1g, 1h Color filter 2a, 2b, 2c, 2d, 2e, 2f, 2g, 2h, 2z Photosensitive cell 3 Silicon substrate 11 Imaging lens 12 Broadband optical filter 13 Imaging Element 13a Imaging unit (imaging surface)
DESCRIPTION OF SYMBOLS 14 Signal generation / reception part 15 Image processing part 16 Image memory 17 Signal output part 20 (Compound eye) imaging lens 21 Monocular black-and-white camera 22 Concave lens array 23 Imaging lens 31 Silicon oxide film

Claims (22)

An imaging device in which a plurality of pixel units are two-dimensionally arranged on the imaging surface, each pixel unit including first to Nth (N is an integer of 2 or more) photosensitive cells, An image sensor that outputs a photoelectric conversion signal corresponding to the amount of received light; and
1st to Nth optical filters disposed respectively facing the first to Nth photosensitive cells, each having one or more transmittance peaks or underpeaks in a predetermined wavelength band And the first to Nth optical filters having different total peak numbers in which the number of peaks and the number of underpeaks are combined,
A signal processing circuit for processing photoelectric conversion signals output from the first to Nth photosensitive cells;
With
The transmittance Y of each of the first to Nth optical filters periodically changes with respect to the change of the variable Z related to the wavelength λ in the predetermined wavelength band,
The signal processing circuit includes N photoelectric conversion signals from the first to Nth photosensitive cells, N types of periodic functions having the independent variable as the variable Z, and the variable for the minute change in the wavelength λ. Calculate the spectral characteristics of the captured image using the change value of Z.
Multispectral camera. The pixel unit may be an unfiltered photosensitive cell that is not arranged corresponding to any of the first to Nth photosensitive cells and the first to Nth optical filters, or the first to Nth photosensitive cells. A transmittance averaged photosensitive cell in which an optical filter having a transmittance intermediate between the peak and the under peak in the optical filter is disposed;
2. The multispectral camera according to claim 1, wherein the signal processing circuit calculates the spectral characteristics by further using a photoelectric conversion signal from the non-filtered photosensitive cell or the transmittance averaged photosensitive cell. 3. A band transmission filter that transmits only light of a predetermined wavelength band;
The signal processing circuit captures spectral characteristics of a captured image calculated using the N photoelectric conversion signals acquired by imaging through the band-pass filter, excluding the first to Nth optical filters. The multispectral camera according to claim 1, wherein the spectral characteristic of the captured image excluding the influence of the imaging optical system is calculated by dividing by the spectral characteristic of the optical system.   Of the first to N-th optical filters, the i-th (i is an integer greater than or equal to 1 and less than or equal to N) optical filter has a total peak number of i + 1, and the N types of periodic functions have a predetermined frequency. 4. The multispectral camera according to claim 1, wherein the multispectral camera is N cosine functions cos (iZ) having any one of an integer multiple of 1 to N times. 5. The signal processing circuit has E (0) as the photoelectric conversion signal of the non-filtered photosensitive cell or the transmittance averaged photosensitive cell, and E (i) as the photoelectric conversion signal of the i-th photosensitive cell.
The result a (0) obtained by multiplying E (0) by the first constant k1 and the result obtained by multiplying E (0) by the third constant are subtracted from the result obtained by multiplying E (i) by the second constant k2. And a (i) cos (iZ) obtained by multiplying the result a (i) obtained by dividing the fourth constant by the cosine function cos (iZ),
By multiplying the result of adding a (0) and all of a (1) cos (Z) to a (N) cos (NZ) by the change value of the variable Z with respect to the minute change of the wavelength λ, The multispectral camera according to claim 2, wherein the spectral characteristic of the captured image is calculated.   The signal processing circuit corrects photoelectric conversion signals from the first to Nth photosensitive cells by multiplying photoelectric conversion signals from the first to Nth photosensitive cells by a fifth constant. The multispectral camera according to claim 5.   In each pixel unit, the first to Nth photosensitive cells are arranged around the non-filtered photosensitive cell or the transmittance averaged photosensitive cell. The multispectral camera according to any one of 6.   The multispectral camera according to claim 7, wherein the signal processing circuit performs signal processing for each pixel unit and generates interpolation information using a photoelectric conversion signal from an adjacent pixel unit.   The multispectral camera according to claim 7 or 8, wherein the imaging device has a plurality of microlenses arranged for each pixel unit.   The signal processing circuit uses data indicating spectral characteristics of a plurality of color filters to perform an integral operation on the calculated function indicating the spectral characteristics of the captured image and the function indicating the spectral characteristics of the color filter. The multispectral camera according to claim 1, which calculates a color signal. A compound eye optical system including N + 1 (N is an integer of 2 or more) optical systems;
1st to Nth optical filters having one or more transmittance peaks or underpeaks in a predetermined wavelength band, and having different total peaks combined with the number of peaks and the number of underpeaks;
An imaging device having first to Nth photosensitive cell groups opposed to the first to Nth optical filters, each of the photosensitive cells outputting a photoelectric conversion signal corresponding to the amount of received light;
A signal processing circuit for processing photoelectric conversion signals output from the first to Nth photosensitive cells;
With
The transmittance Y of each of the first to Nth optical filters periodically changes with respect to the change of the variable Z related to the wavelength λ in the predetermined wavelength band,
The first to Nth optical filters are respectively arranged to face N optical systems of the N + 1 optical systems in the compound eye optical system,
The image pickup device further includes an unfiltered photosensitive cell group in which none of the first to Nth optical filters are arranged to face each other, or an intermediate between a peak and an under peak in the first to Nth optical filters. A transmittance averaged photosensitive cell group disposed oppositely with an optical filter having a transmittance of
The signal processing circuit includes N types of photoelectric conversion signals from the first to Nth photosensitive cell groups, a photoelectric conversion signal from the non-filter photosensitive cell group, or the transmittance averaged photosensitive cell group. A spectral characteristic of the captured image is calculated using a photoelectric conversion signal of N, a periodic function of N types having an independent variable as the variable Z, and a change value of the variable Z with respect to a minute change of the wavelength λ.
Multispectral camera. A band transmission filter that transmits only light of a predetermined wavelength band;
The signal processing circuit has the first to Nth optical filters for the spectral characteristics of the captured image calculated using the N types of photoelectric conversion signals acquired by imaging through the band pass filter. The multispectral camera according to claim 11, wherein when a transmittance peak or under peak attenuates or increases with respect to a wavelength change, a spectral characteristic of a captured image in which the attenuation or increase is corrected is calculated. A band transmission filter that transmits only light of a predetermined wavelength band;
The signal processing circuit captures spectral characteristics of a captured image calculated using the N types of photoelectric conversion signals acquired by imaging through the band-pass filter, excluding the first to Nth optical filters. The multispectral camera according to claim 11 or 12, wherein a spectral characteristic of a captured image excluding the influence of the imaging optical system is calculated by dividing by a spectral characteristic of the optical system.   Of the first to N-th optical filters, the i-th (i is an integer greater than or equal to 1 and less than or equal to N) optical filter has a total peak number of i + 1, and the N types of periodic functions have a predetermined frequency. 4. The multispectral camera according to claim 1, wherein the multispectral camera is N cosine functions cos (iZ) having any one of an integer multiple of 1 to N times. 5. The signal processing circuit has E (0) as the photoelectric conversion signal of the unfiltered photosensitive cell, and E (i) as the photoelectric conversion signal of the i-th photosensitive cell.
The result a (0) obtained by multiplying E (0) by the first constant k1 and the result obtained by multiplying E (0) by the third constant are subtracted from the result obtained by multiplying E (i) by the second constant k2. And a (i) cos (iZ) obtained by multiplying the result a (i) obtained by dividing the fourth constant by the cosine function cos (iZ),
By multiplying the result of adding a (0) and all of a (1) cos (Z) to a (N) cos (NZ) by the change value of the variable Z with respect to the minute change of the wavelength λ, The multispectral camera according to claim 11, wherein the spectral characteristic of the captured image is calculated.   The signal processing circuit corrects photoelectric conversion signals from the first to Nth photosensitive cells by multiplying photoelectric conversion signals from the first to Nth photosensitive cells by a fifth constant. The multispectral camera according to claim 15.   The multi-spectrum according to any one of claims 11 to 16, wherein the signal processing circuit calculates spectral characteristics of the captured image after correcting distortion of an image formed by each optical system in the compound-eye optical system. camera.   The multispectral camera according to claim 11, wherein the compound eye optical system includes a plurality of convex lenses.   The multispectral camera according to claim 18, wherein the plurality of convex lenses includes a decentered convex lens.   The multispectral camera according to claim 11, wherein the compound-eye optical system includes a plurality of concave lenses and one convex lens.   The multispectral camera of claim 20, wherein the plurality of concave lenses includes a decentered concave lens.   The signal processing circuit uses data indicating spectral characteristics of a plurality of color filters to perform an integral operation on the calculated function indicating the spectral characteristics of the captured image and the function indicating the spectral characteristics of the color filter. The multispectral camera according to claim 11, which calculates a color signal.

Images