JP2016144185A - Imaging apparatus and imaging system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a novel imaging apparatus which acquires spectral information of an imaging object by disposing a band pass filter for color separation at a position of a diaphragm of an image formation lens.SOLUTION: The imaging apparatus comprises: an image formation lens system 10; a two-dimensional microlens array 20 that is disposed at an image side of the image formation lens system; a two-dimensional imaging element 30 which is disposed at an image side of the microlens array and performs photoelectric conversion in optical information that is incident via the microlens array; and a band pass filter FL that is disposed while being substantially matched with a diaphragm position of the image formation lens system 10. The band pass filter FL is formed by arraying two or more kinds of slave filters FR, FY, FG and FB of which the spectral transmission properties are separated mutually, in mutually orthogonal directions X and Y cyclically for two or more periods in such a manner that the slave filters of the same type are not neighboring to each other in the directions. An image of the band pass filter FL is substantially formed on a light-receiving surface of the imaging element 10.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an imaging apparatus and an imaging system.

  A plenoptic camera has been put into practical use and is intended to be applied to various uses such as image capturing, distance measurement, and spectroscopic measurement.

  For example, Patent Documents 1 and 2 disclose an invention in which a plenoptic camera configuration is used, a band-pass filter is disposed at the aperture position of the imaging lens, and spectral information of a subject to be photographed is acquired.

  An object of the present invention is to realize a novel image pickup apparatus that obtains spectral information of a subject to be photographed by disposing a band-pass filter for color separation at the position of the stop of the imaging lens.

  An imaging device according to the present invention includes an imaging lens system, a two-dimensional microlens array disposed on the image side of the imaging lens system, an image side of the microlens array, and the microlens array via the microlens array. A two-dimensional image sensor that photoelectrically converts incident light information and a band-pass filter that is arranged substantially in accordance with the aperture position of the imaging lens system, the band-pass filter having spectral transmission characteristics Two or more types of child filters separated from each other are arranged in two or more cycles in a direction orthogonal to each other so that the same type of child filters are not adjacent to each other in the direction. The image is substantially formed on the light receiving surface of the image sensor.

  According to the present invention, it is possible to realize a novel imaging apparatus that obtains spectral information of a subject to be photographed by disposing a bandpass filter for color separation at the aperture position of the imaging lens.

It is a figure for demonstrating one Embodiment of an imaging system. It is a figure for demonstrating the spectral transmission characteristic of a band pass filter. It is a figure for demonstrating an example of a band pass filter. It is a figure for demonstrating the mode of the image formation on the light-receiving surface of an image pick-up element in the case of using the band pass filter of FIG. It is a figure which shows the filter image IMFLij imaged on the macro lens which belongs to the micro lens MLij and each micro lens. It is a figure for demonstrating the process of obtaining spectral information. It is a figure which shows another example of a band pass filter. It is a figure which shows the other example of a band pass filter.

  Hereinafter, embodiments will be described.

FIG. 1 is a diagram for explaining one embodiment of an imaging system of the present invention.
In FIG. 1A, reference numeral 10 denotes an “imaging lens system”, reference numeral 20 denotes a “two-dimensional microlens array”, reference numeral 30 denotes a “two-dimensional imaging device”, reference numeral 40 denotes an “image processing unit”, Each of them is shown, and symbol FL indicates a “bandpass filter”.
FIG. 1A schematically illustrates the imaging system for the sake of simplicity.
The imaging lens system 10 is depicted as a simple lens. The imaging lens 10 can be composed of a single lens, but is generally composed of a plurality of lenses.
The band pass filter FL is disposed at the position of the stop of the imaging lens system 10. In the drawing, the band pass filter FL is drawn so as to be located in the lens of the imaging lens system 10, but this is for the sake of simplification of the drawing. When the imaging lens 10 is composed of two or more lenses, the position where the diaphragm is arranged is the position between the lenses.

A band pass filter FL is arranged at the aperture of the diaphragm arranged in this way.
The two-dimensional microlens array 20 is a microlens that is a microlens arranged in a two-dimensional array, and the arranged microlenses have the same shape.
The microlens array 20 is disposed so as to substantially match a predetermined position on the image side of the imaging lens system 10, for example, an image side focal plane.
When the microlens array 20 is arranged at the image side focal plane position of the imaging lens system 10, an image of the subject at a certain object distance by the imaging lens system 10 is near the microlens array surface of the microlens array 20. Form an image.

The two-dimensional image sensor 30 is a so-called area sensor in which minute light receiving elements are two-dimensionally arrayed. Then, the micro lens array 20 is disposed in the vicinity of the micro lens array 20 on the side opposite to the imaging lens system 10 via the micro lens array 20.
The image sensor 30 is arranged such that the light receiving surface is located at a position substantially conjugate with the band pass filter FL. The image sensor 30 is “a monochrome sensor in which a color filter for each pixel is not mounted”. Of the light emitted from the subject, the light beam incident on the imaging lens system 10 is the “spectral energy measurement target”.
That is, color separation for spectral energy measurement is performed exclusively by a band pass filter.

  As described above, the imaging lens system 10 is generally composed of a plurality of lenses.

Therefore, the image of the band pass filter FL is substantially formed on the light receiving surface by the action of the “lens closer to the micro lens array than the band pass filter FL” and the micro lens among the plurality of lenses. To do.
Since this image formation occurs for each microlens, a “bandpass filter image” for each microlens is formed on the light receiving surface of the image sensor 30.

The light receiving elements arranged two-dimensionally on the light receiving surface of the image sensor 30 correspond to a plurality of light receiving elements for each microlens. For example, an array of n ² light receiving elements corresponds to a single microlens, where n (≧ 1) is a natural number, such as 4 (2 × 2) or 9 (3 × 3).
In this way, a two-dimensional array of a plurality of light receiving elements corresponding to one microlens is called a “macro pixel”.
Considering an arbitrary microlens, all the “light rays that form an image of a bandpass filter” incident on the microlens are incident on a macropixel corresponding to the microlens.
At this time, the incident directions of the light rays incident on the individual light receiving elements constituting the macro pixel are different from each other. Therefore, the output of the image sensor 30 includes “intensity and information on the incident direction” of light rays that enter the light receiving element from the subject.

  Each macro pixel corresponding to each micro lens in the micro lens array will be referred to as a “macro pixel belonging to a micro lens”.

The image processing unit 40 is configured as a computer, a CPU, or the like.
In general, the bandpass filter FL is configured such that “two or more types of child filters whose spectral transmission characteristics are separated from each other” are arranged in a direction orthogonal to each other and “two or more periods cyclically so that the same type of child filters are not adjacent to each other”. Array ".

  Each of the sub-filters transmits “light of a predetermined wavelength band”. However, at the base portion of the band to transmit, the sub-filters of different types may overlap each other.

  In the embodiment shown in FIG. 1, the band-pass filter FL is configured by a combination of four types of child filters FB, FG, FY, and FR.

  The wavelength bands transmitted by the child filters FB, FG, FY, and FR can be appropriately selected as long as they are different from each other.

In the following, it is assumed that the child filter FB has “spectral transmission characteristics that mainly transmit light in the blue (B) wavelength region” for the sake of description.
The sub-filters FG, FY, FR have spectral transmission characteristics that transmit “mainly light in the green (G) wavelength region, yellow (Y) wavelength region, and red (R) wavelength region”, respectively.

FIG. 2 schematically shows examples of these four types of spectral transmission characteristics TB, TG, TY, and TR as explanatory diagrams.
Returning to FIG. 1, the child filters FB, FG, FY, FR of the bandpass filter FL are arranged in two directions (X direction and Y direction) orthogonal to each other as shown in FIG. Three cycles are arranged cyclically so that child filters of the same type are not adjacent.
That is, in the array of child filters, the child filters adjacent to each other in the X and Y directions are of different types. The arrangement of the two types of child filters is cyclic (periodic repetition) in the X direction and the Y direction.

That is, in the embodiment of the imaging system shown in FIG. 1, the “imaging device portion” includes the imaging lens system 10, the microlens array 20, the imaging element 30, and the band-pass filter FL.
The microlens array 20 is a microarray of microlenses arranged two-dimensionally, and is disposed on the image side of the imaging lens system 10.
The imaging element 30 is a two-dimensional array of minute light receiving elements, and is arranged on the image side of the microlens array 20.

The band pass filter FL is arranged so as to “substantially match the aperture position” of the imaging lens system 10. The band-pass filter FL has two or more types of child filters FB, FG, FY, and FR whose spectral transmission characteristics are separated from each other.
These child filters are cyclically arranged in two or more periods in the X and Y directions orthogonal to each other so that the same type of child filters are not adjacent to each other in that direction.
Then, the image of the bandpass filter FL is substantially formed on the light receiving surface of the image sensor 30.

FIG. 3 shows the bandpass filter FL shown in FIG.
“Individual child filters” arranged in a two-dimensional manner are distinguished by reference numerals as shown in the figure.
“B00 to B22” indicate nine child filters FB arranged two-dimensionally.
“G01 to G23” indicate nine child filters FG arranged in two dimensions.
“Y10 to Y32” indicate nine child filters FY arranged in two dimensions.

  “R11 to R33” indicate nine child filters FR arranged two-dimensionally.

  As described above, a “bandpass filter image” is formed on the light receiving surface of the image sensor 30 for each microlens.

FIG. 4 is a diagram for explaining this imaging. As the bandpass filter, the bandpass filter FL described with reference to FIG. 3 is taken as an example.
In FIG. 4A, reference numeral ML indicates one of “microlenses” constituting the microlens array, and reference numeral IMFL indicates an image (hereinafter referred to as “filter image”) of the bandpass filter FL formed by the microlens ML. .).
The two-dimensional array of light receiving elements that receive the filter image IMFL is the aforementioned “macro pixel”.
In the filter image IMFL shown in FIG. 4 (a), “the part given the same reference sign” as the reference sign given to the child filter of the bandpass filter FL shown in FIG. 3 is the image of the child filter given the same reference sign. It is. For example, the portion indicated by “reference B00” in the filter image IMFL is an image of the child filter B00 in FIG.

  These images corresponding to the individual child filters are called child filter images, and the code is, for example, the child filter image B00.

The filter image IMFL is substantially imaged on the light receiving surface of the image sensor, and “one or more of the light receiving elements constituting the light receiving surface” corresponds to each child filter image B00 and the like of the filter image IMFL.
In this correspondence, the child filter image and the light receiving element may correspond 1: 1, or N (plurality) of light receiving elements may correspond to one child filter image.
When the correspondence between the light receiving element and the child filter image is 1: 1, the output value of the light receiving element corresponding to one child filter image corresponds to the spectral energy intensity of the child filter image.
The output value of the light receiving element is proportional to “the product of the light energy of the light beam transmitted through the child filter out of the light emitted from the subject and the spectral sensitivity of the light receiving element”. Since the spectral sensitivity of the light receiving element is known at the time of design, the spectral energy intensity can be obtained by dividing the output value by the spectral sensitivity.

When “N (≧ 1) light receiving elements correspond to one child filter image” corresponds to each macro pixel, if N ≠ 1, “average value of output values of N light receiving elements” is obtained. The value divided by the spectral sensitivity can be defined as “the spectral energy intensity of the child filter image in the macro pixel”.
Alternatively, a value obtained by dividing one of the output values of the N light receiving elements as a representative value and divided by the spectral sensitivity can be set as “the spectral energy intensity of the child filter image in the macro pixel”.
Of course, if N = 1, one light receiving element corresponds to each child filter image. In this case, the output value of the light receiving element divided by the spectral sensitivity is “the child filter in the macro pixel”. The spectral energy intensity of the image ".
The ratio between the size of the microlens (lens diameter) and the size of the light receiving element (light receiving surface) of the image sensor (light receiving surface diameter) is generally in the range of 30: 1 to 2: 1.
For example, in the example of FIG. 4, consider a case where the ratio of the lens diameter of the microlens ML to the light receiving surface diameter of the light receiving element of the imaging element is “30: 1”.
In this case, since the size of the child filter image B00 and the like is 1/6 of the lens system of the micro lens, 5 × 5 = 25 light-receiving elements correspond to one child filter image and a macro pixel is obtained. Will be composed.

In FIG. 4B, among the child filter images B00 to R33 constituting the filter image IMFL, “the child filter images in the uppermost row, the lowermost row, the leftmost row, and the rightmost row” in FIG. This is a child filter image formed on the periphery of the image area, and image degradation is likely to occur.
Further, noise due to “stray light of adjacent microlenses” is likely to be generated in the image forming regions in the peripheral portions.
Therefore, the “light reception output” obtained by these child filter images is ignored, and the output value obtained by the “4 × 4 child filter images R11 to B22 shown in the high density diagram” in the figure is regarded as “effective output”. Use.

  Hereinafter, in the imaging system illustrated in FIG. 1, a process of obtaining “spectral information corresponding to various child filters” from the output of the imaging element 30 by the image processing unit 40 will be described.

As the band-pass filter FL, it is assumed that the above-described four types of child filters FB, FG, FY, and FR are “6 × 6 array”.
In the macro pixel, it is assumed that “one light receiving element corresponds to each child filter image” in the filter image IMFL. That is, one micropixel is formed of 6 × 6 light receiving elements.
The “number of macro pixels” is determined according to the size of the light receiving surface of the image sensor and the number and size of the micro lenses in the micro lens array 20, but here “six micro lenses are arranged in two rows and three columns”. A simplified explanation will be given.

FIG. 5 shows these six microlenses MLij (i = 1 to 3, j = 1 to 2) and filter images IMLij (i = 1 to 3, j = images formed on macro pixels belonging to the individual microlenses MLij. 1-2).
In the example in the description, as described above, since “one light receiving element corresponds to each child filter image” in the filter image IMFL, each macro pixel is composed of 36 light receiving elements arranged in 6 rows and 6 columns. It is configured.
Out of the outputs of these 36 light receiving elements, the outputs from 16 light receiving elements corresponding to the central “4 × 4 child filter images R11 to B22 shown in the high density image” are effective outputs.

Acquisition of spectral information by the image processing unit 40 is performed as follows.
A “spectral image” is generated in accordance with each of the child filter colors: R, G, B, and Y.
FIG. 6 shows these four spectral images.
The macro pixels belonging to the microlens MLij (i = 1 to 3, j = 1 to 2) in FIG. 5 are represented by reference symbols MPij (i = 1 to 3, j = 1 to 2).

Among the macro pixels MPij (i = 1 to 3, j = 1 to 2), a spectral image by an effective output of the light receiving element on which an R color child filter image is formed is denoted by a symbol MPijR.
A spectral image based on the effective output of the light receiving element on which the G-color child filter image is formed is indicated by a symbol MPijG.

A spectral image based on the effective output of the light receiving element on which the Y-color child filter image is formed is indicated by a symbol MPijY, and a spectral image based on the effective output of the light receiving element on which the B-color child filter image is formed is indicated by a symbol MPijB.
FIG. 6A shows an R-color monochromatic spectral image (consisting of spectral images MP11R to MP23R of six macro pixels MP11 to MP23).
In this figure, for example, “R11” represents the output of the light receiving element on which the child filter image R11 is formed.
Similarly, FIGS. 6B to 6D show single color spectral images of G color, Y color, and B color, respectively. In this way, single-color spectral images of four colors R, G, Y, and B can be acquired as “spectral information”.
With these four color monochromatic spectral images, the spectral characteristics of the imaged subject can be known.

  Further, if these four color monochromatic spectral images are superimposed and synthesized, a color image of the subject can be obtained.

  Above, a band-pass filter having four color child filters FR, FY, FG, and FB of red (R), yellow (Y), green (G), and blue (B) has been described.

  However, the “bandpass region (transmission wavelength region) of a plurality of types of child filters” combined to form the bandpass filter is not limited to the above example.

  A plurality of types of child filters whose spectral transmission characteristics are separated from each other can be combined.

  Also, the types of child filters to be combined to form the bandpass filter are not limited to four, and two or three types of child filters, or five or more types of child filters can be combined.

  The plurality of types of child filters constituting the bandpass filter are arranged in two or more cycles in a direction orthogonal to each other so that the same type of child filters are not adjacent to each other in the direction. The image is substantially formed on the light receiving surface of the image sensor.

Accordingly, at least 16 child filter images are formed on “macropixels belonging to individual microlenses” of the microlens array.
For this reason, the number of child filters imaged on the macro pixel is large, and the “light reception output” obtained from the child filter image at the periphery of the filter image is ignored. The output value of the light receiving element to be used can be used as “effective output”.

  In addition, the fact that the number of child filters imaged on the macro pixel is large means that the individual child filter images to be imaged are small, so that the resolution at each macro pixel becomes high and spectral information with high resolution can be obtained. Can do.

7 and 8 show two examples of other bandpass filters.
The band-pass filter FL2 shown in FIG. 7 has two types of child filters FS and FT (these spectral transmission characteristics are separated from each other) as shown in FIG. 7A, as shown in FIG. 7B. In addition, three cycles are alternately arranged in the X direction and the Y direction.

  By using such a bandpass filter FL2, it is possible to obtain “subject spectral information” corresponding to the spectral transmission characteristics of the child filters FS and FT.

  The band-pass filter FL3 shown in FIG. 8 has three types of child filters FV, FW, and FZ (these spectral transmission characteristics are separated from each other) as shown in FIG. As shown, three cycles are alternately arranged in the X direction and the Y direction.

  By using such a band-pass filter FL3, “subject spectral information” corresponding to the spectral transmission characteristics of the child filters FV, FW, and FZ can be obtained.

  Of course, in these examples, it goes without saying that the cyclic arrangement of the child filters is not limited to three periods.

  The bandpass filter can be manufactured by the following method.

  For example, this is a method of realizing the spectral transmission characteristics of the child filters constituting the band-pass filter as a well-known “dielectric multilayer film”.

  Another example is a method of forming a child filter by holding a pigment in a transparent body such as glass or plastic.

  The bandpass filter preferably has a spectral transmission characteristic that does not depend on the incident angle. From this point of view, a bandpass filter in which a child filter is formed by holding a pigment in a transparent material such as glass or plastic has a spectral transmission characteristic. Is preferable because it hardly depends on the incident angle.

  The one that realizes the spectral transmission characteristics of the child filter by the dielectric multilayer film is a diffraction phenomenon due to reflection inside the multilayer film, and some incident angle dependence occurs. However, such incident angle dependence is a problem in practical use. It is possible to suppress to such an extent that it does not become.

  As described above, according to the present invention, the following imaging device and imaging system can be realized.

[1]
Imaging lens system 10, two-dimensional microlens array 20 disposed on the image side of the imaging lens system, and light disposed on the image side of the microlens array and incident via the microlens array 20 A two-dimensional image sensor 30 for photoelectrically converting information; and a band-pass filter FL arranged substantially in accordance with the aperture position of the imaging lens system 10, wherein the band-pass filter FL has a spectral transmission characteristic. Two or more types of child filters separated from each other are arranged in two or more cycles cyclically in the X and Y directions orthogonal to each other so that the same type of child filters are not adjacent to each other in the directions. An image pickup apparatus in which an image of FL is substantially formed on the light receiving surface of the image pickup device 30.

[2]
[1] The imaging apparatus according to [1], wherein the band-pass filter FL2 includes two types of child filters FS and FT whose spectral transmission characteristics are separated from each other.

[3]
[1] The imaging apparatus according to [1], wherein the bandpass filter FL3 includes three types of child filters FV, FW, and FZ whose spectral transmission characteristics are separated from each other.

[4]
[1] The imaging apparatus according to [1], wherein the band-pass filter FL includes four types of child filters FB, FG, FY, and FR whose spectral transmission characteristics are separated from each other.

[5]
The imaging device according to any one of [1] to [4], wherein the bandpass filter is a plate-like transparent body that holds pigments of different colors according to the arrangement of the child filters.

[6]
In the imaging device according to any one of [1] to [5], n (≧ 1) is a natural number and n ² pieces of individual child filter images that are substantially imaged on the light receiving surface of the light receiving element. An imaging device corresponding to the arrangement of the light receiving elements.

[7]
An imaging system including the imaging device according to any one of [1] to [5] and an image processing unit 40, wherein the image processing unit 40 receives various kinds of signals from an output of the imaging device 30 of the imaging device. An imaging system for obtaining spectral information corresponding to a child filter.

The preferred embodiments of the present invention have been described above. However, the present invention is not limited to the specific embodiments described above, and the invention described in the claims unless otherwise specified in the above description. Various modifications and changes are possible within the scope of the above.
The effects described in the embodiments of the present invention are merely a list of suitable effects resulting from the invention, and the effects of the present invention are not limited to those described in the embodiments.

10 Imaging optical system
20 Two-dimensional microlens array
30 Two-dimensional image sensor
FL bandpass filter
FB, FG, FY, FR child filter
40 Image processing unit

JP 2014-75780 A JP 2004-89075 A

Claims (7)

An imaging lens system;
A two-dimensional microlens array disposed on the image side of the imaging lens system;
A two-dimensional image sensor that is arranged on the image side of the microlens array and photoelectrically converts light information incident through the microlens array;
A band-pass filter disposed substantially coincident with the aperture position of the imaging lens system,
The bandpass filter is formed by arranging two or more types of child filters whose spectral transmission characteristics are separated from each other in a direction orthogonal to each other in two or more cycles so that the same type of child filters are not adjacent to each other in the direction. Become
An imaging apparatus in which an image of the bandpass filter is substantially formed on a light receiving surface of the imaging element. The imaging apparatus according to claim 1,
An imaging apparatus in which the bandpass filter has two types of child filters whose spectral transmission characteristics are separated from each other. The imaging apparatus according to claim 1,
An imaging apparatus in which the bandpass filter has three types of child filters whose spectral transmission characteristics are separated from each other. The imaging apparatus according to claim 1,
An imaging apparatus in which the bandpass filter has four types of child filters whose spectral transmission characteristics are separated from each other. The imaging apparatus according to any one of claims 1 to 4, wherein
An imaging device in which the bandpass filter is a plate-like transparent body that holds pigments of different colors according to the arrangement of the child filters. The imaging apparatus according to any one of claims 1 to 5,
An imaging device in which an array of n ² light receiving elements corresponds to each child filter image substantially formed on the light receiving surface of the light receiving element, where n (≧ 1) is a natural number. An imaging system comprising the imaging device according to any one of claims 1 to 6 and an image processing unit,
An imaging system in which the image processing unit obtains spectral information corresponding to various child filters from an output of an imaging element of an imaging device.

Images