JP2011182237A - Compound-eye imaging device, and image processing method in the same - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To improve an accuracy of distance measurement in a compound-eye imaging device for obtaining a plurality of spectroscopic images. <P>SOLUTION: One of n-types of optical filters 7 is arranged for every facet unit 9 including a minute optical lens 5. As the n-types of optical filters 7, filters each having a characteristic where wavelengths or wavelength bands permeating in common on a transmission spectrum are present, are selected for optional two types of optical filters. Accordingly, since a part appearing in common in two optional facet images in a compound-eye image are present when a general object is imaged, parallax information can be extracted from the common part. Since the accuracy of distance measurement depends on the amount of parallax information, the distance measurement accuracy is improved by increasing certainty of acquisition of parallax information. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a compound-eye imaging apparatus that acquires a plurality of object images derived from a single object using a plurality of imaging optical systems, and an image processing method that processes data obtained by the apparatus.

  In various fields such as electronic commerce, electronic archiving, and remote pathological diagnosis, it is important to acquire images with high color reproducibility. A conventional general imaging apparatus uses three primary colors, and is insufficient to reproduce the faithful color of the subject. On the other hand, an imaging device called a multispectral camera that can obtain an image from a subject as an image of many primary colors has been recently proposed. A multispectral camera obtains a multi-wavelength spectral image of a subject, and several methods are known, and one of them uses a compound eye imaging device.

  The compound-eye imaging device captures a plurality of single-eye images of an optical lens array in which a plurality of micro optical lenses are two-dimensionally arranged and a single subject formed by each optical lens of the optical lens array. This is a combination of a solid-state image sensor (see Non-Patent Document 1, etc.). Since each optical lens of the optical lens array is two-dimensionally distributed on the plane, each single-eye image formed on the detection surface of the solid-state image sensor is an image viewed from a slightly different angle with respect to the subject. (Image having parallax). For this reason, a compound eye image obtained by collecting single-eye images includes information on high resolution in addition to information on the three-dimensional shape of the subject. Therefore, by using a large number of individual images slightly different from each other, it is possible to generate a reconstructed image based on a larger amount of information, and it is possible to obtain a reconstructed image having a higher definition than the resolution of each individual eye image. .

  As described in Patent Document 1, for example, when the compound-eye imaging device is used in a multispectral camera, a band-pass filter (color filter) having a different transmission wavelength for each lens is installed in front of the optical lens array. Thereby, each single eye image has specific wavelength information. By performing predetermined data processing on an image signal based on a compound eye image having information of such a specific wavelength, a distance is estimated for each minute region on the subject, and image reconstruction processing is performed using the distance information. Thus, a plurality of spectral images having specific wavelength information can be obtained.

  However, in the multispectral camera using the above conventional compound eye imaging device, depending on the reflection spectral characteristics of the subject, there is no correlation between the single eye images that have passed through different bandpass filters. There may be no common part between the images. In such a case, sufficient correction cannot be performed when correcting the positional deviation between images due to parallax, and distance information cannot be obtained accurately, so that the accuracy of image reconstruction may be reduced. In addition, since the light coming from the subject is greatly attenuated when passing through the bandpass filter, the amount of light incident on the solid-state imaging device is reduced, and particularly when the real-time imaging is performed, the S / N of the image is low. There is also.

  In addition, there are the following problems. In a compound eye imaging device, a CMOS sensor is often used for a solid-state imaging device. The CMOS sensor can read out pixel signals from the light receiving elements in each row with a time difference for each row by using a function called a rolling shutter. When this rolling shutter function is used in a compound-eye imaging device, a series of single-eye images having a time difference in a compound-eye image can be obtained. Patent Document 2 describes that the movement of an object is detected from a single compound eye image using this. Patent Document 3 describes that a subject is sequentially irradiated with light having a plurality of types of slit patterns having different phases, and a three-dimensional shape of the subject is measured using a compound eye image obtained correspondingly. ing.

  If the time difference between a series of single-eye images is constant, the time-frequency component has a zero point, and in that case, accurate spatial (position) information cannot be obtained, and the accuracy of motion detection and three-dimensional shape measurement may be reduced. There is.

  In addition, as described in Patent Documents 4 and 5 and the like, various methods have been proposed for reconstructing a single high-resolution image from a plurality of low-resolution single-eye images. The purpose is exclusively to increase the spatial resolution, and it is not considered to reconstruct a spectral image having a high wavelength resolution based on a plurality of single-eye images having a specific wavelength spectrum.

  In addition, since the compound-eye imaging apparatus uses a lens array in which a large number of optical lenses are arranged, it is difficult to introduce an autofocus function for mechanically moving the lens to find out a focus position. In general, a compound-eye imaging device has a deeper depth of focus than a monocular imaging device, and the distance range for focusing is relatively wide, but the depth of field can be further expanded in order to expand the range that can be measured. Necessary.

JP 2003-163938 A JP 2008-244649 A JP 2008-224629 A JP 2003-141529 A JP 2005-167484 A

Supervised by Satoshi Ota, “Latest Trends in CMOS Image Sensors—From High Performance and High Functionality to Application Deployment”, “Chapter 13 Compound Eye Imaging System-TOMBO” (by Jun Tanida), CM Publishing, 2007 4 Moon

  The present invention has been made in view of the above problems, and in a compound eye imaging device intended to improve color reproducibility, it is possible to correct a single image from a plurality of images by accurately correcting misalignment between individual images due to parallax. The main purpose is to improve the accuracy when one image is reconstructed.

  Another object of the image processing method of the compound-eye imaging apparatus according to the present invention is to perform image computation by appropriately setting the time difference when a plurality of single-eye images included in one compound-eye image have time differences. In addition to improving the accuracy of processing and reconstructing a single image from a compound eye image, the accuracy of motion detection and three-dimensional shape measurement is improved.

  Another object of the image processing method of the compound eye imaging device according to the present invention is to reconstruct a spectral image having a high wavelength resolution based on a plurality of single-eye images having a specific wavelength spectrum by relatively simple processing. That is.

  Another object of the image processing method of the compound-eye imaging device according to the present invention is to enlarge the depth of field without using a mechanical drive system such as autofocus, and to expand the applicable distance range of the distance measurement to obtain a compound eye image. To improve the accuracy when reconstructing a single image.

In order to achieve the above object, a compound eye imaging device according to a first invention comprises a lens array in which m (m is an integer of 4 or more) micro optical lenses are two-dimensionally arranged, and a plurality of micro light receiving devices. A solid-state imaging device having a two-dimensionally arranged portion, and forming a single-eye image of a subject by each of the micro optical lenses in different areas on the light-receiving surface of the solid-state imaging device. In a compound eye imaging device that acquires an image signal corresponding to a single eye image and generates a single or a plurality of reconstructed images from the image signal,
Optical filters having a predetermined wavelength transmission characteristic are arranged corresponding to all or a part of the m micro optical lenses, and the maximum m optical filters have different wavelength transmission characteristics n (n is 2). It is characterized in that it includes at least one of the above integer) types of optical filters, and any two of the n types of optical filters are correlated with respect to wavelength.

  A state in which two optical filters are “correlated with respect to wavelength” means that the two optical filters have substantially non-zero transmittance at a certain wavelength or a certain wavelength band, in other words, It means that there is a wavelength or wavelength band that is transmitted in common.

  According to the compound-eye imaging device of the present invention, when photographing a general subject, at least a part of the single-eye images that have passed through two different types of optical filters among the n types of optical filters are shared. A part appears. The presence of a common part on a single image means that disparity information can be extracted. Therefore, except for very special subjects, normally, parallax information can be extracted from any combination of single-eye images, and the distance for each minute region on the subject is calculated with high accuracy using the parallax information. be able to. By increasing the accuracy of the distance information, the accuracy of reconstruction of an image (spectral image) using the distance information is also increased.

  Basically, the optical filters are arranged corresponding to all of the m micro optical lenses, but there may be micro optical lenses in which no optical filter is arranged in part. Since the single-eye image obtained corresponding to the micro optical lens in which no optical filter is arranged in this way includes all luminance information regardless of the color of the subject, for example, this single-eye image is used as a reference single-eye image. be able to. Further, a different optical filter may be arranged for each pixel of the solid-state imaging device, and a substantial filter type may be increased by combining with a color filter arranged in the micro optical lens. Further, an ND (neutral density) filter may be attached in order to compensate for a light amount difference between a single eye provided with an optical filter and a single eye provided with no optical filter.

  In the compound eye imaging device according to the first aspect of the present invention, it is preferable to collect parallax information from various directions in a balanced manner. Therefore, the same number or at most one of the n types of optical filters for the m micro optical lenses, respectively. It is good to allocate by the number difference.

  In addition, since it is not preferable that the parallax information is shifted with respect to a specific wavelength or wavelength band, two adjacent ones of the same type in each dimension of the optical filter arranged in a two-dimensional manner corresponding to the micro optical lens. It is preferable that the optical filters of the same type are dispersedly arranged so that the intervals of the optical filters are all possible intervals. Here, “the interval between two adjacent optical filters of the same type” means that two optical filters of the same type are adjacent to each other without sandwiching another type of optical filter in one dimension. Is “1” as the minimum unit, and is an integral multiple of this minimum unit.

  In addition, when the central single-eye image in the two-dimensional single-eye image is considered as the reference single-eye image, the parallax with the reference single-eye image increases as the single-eye image located on the outer peripheral side increases, Accuracy is improved. Therefore, when there are n types of optical filters that are important for wavelength transmission characteristics and those that are not so, it is better to preferentially assign the former to the outer peripheral side.

In addition, a compound eye imaging device according to the second invention made to achieve the above object includes a lens array in which a plurality of minute optical lenses are two-dimensionally arranged, and a plurality of minute light receiving portions are two-dimensionally arranged. A single solid-state image sensor, and a single-eye image of a subject imaged by each micro-optical lens of the lens array is formed on different areas on the light-receiving surface of the solid-state image sensor. In a compound eye imaging device that sequentially reads an image with a time difference by a rolling shutter and generates a reconstructed image from the read image signal,
By disposing the plurality of micro optical lenses by shifting each micro optical lens independently from each other in a direction parallel to the two-dimensional plane from a position on a lattice point having an equal pitch in the row direction and the column direction, A feature is that an independent time difference is given to the corresponding pixels of the image depending on the combination of the single eye images.

  Note that the row direction and the column direction of the arrangement of the micro optical lenses in the lens array do not have to coincide with the row direction and the column direction of the arrangement of the micro light receiving parts in the solid-state imaging device. By appropriately determining the angular deviation between the two in the row direction, for example, the time delay of each eye image can be made substantially constant when exposure and signal readout are performed by the rolling shutter. At this time, in the compound eye imaging device according to the second aspect of the present invention, since the micro optical lens is not completely arranged on the lattice point and is intentionally displaced, two arbitrary eye images are captured. It is unlikely that the time differences when selected will match. Therefore, the time interval between corresponding pixels does not become constant, and high-precision computation can be performed when computation processing such as image reconstruction is performed based on information of a compound eye image. As a result, a reconstructed image with higher definition than before can be obtained.

A compound eye imaging device according to a third aspect of the invention made to achieve the above object includes a lens array in which a plurality of minute optical lenses are two-dimensionally arranged, and a plurality of minute light receiving portions are two-dimensionally arranged. A solid-state image sensor, and a single-eye image of a subject imaged by each micro-optical lens of the lens array is formed in different areas on the light-receiving surface of the solid-state image sensor, and m In a compound eye imaging device that acquires an image signal corresponding to a single eye image and generates a reconstructed image from the image signal,
By giving aberration to each micro optical lens of the lens array, the distance between the device and the subject is within a predetermined range, and the point spread function for the same image height is rotated with respect to the optical axis regardless of the distance. It is characterized by being arranged symmetrically.

  In other words, this gives an image that is intentionally degraded by the imaging optical system, that is, a blurred image. However, if the point spread function for the same image height is prepared, the blur does not depend on defocusing. Substantially deep depth of field can be obtained by signal processing using the theory of wavefront coding. As a result, the range of applicable distance measurement can be expanded without using mechanical autofocus and the like, so that the apparatus configuration is simplified.

  Of course, the compound-eye imaging devices according to the first to third inventions can be used alone or in combination as appropriate.

A fourth invention made to achieve the above object is an image processing method for reconstructing a spectral image of a subject based on a plurality of single-eye images in the compound-eye imaging apparatus according to the first invention,
Assuming a reconstructed image plane obtained by adding the type identification information and time information of the optical filter to each coordinate on the virtual object plane assuming the separation distance from the light receiving surface of the solid-state imaging device, each pixel of each eye image The pixel value is geometrically plotted on the reconstructed image plane based on the pixel coordinates of the pixel on the light receiving surface, the type of optical filter assigned to the single eye image, and the accumulation start time at the solid-state imaging device at the pixel. A backprojecting step for performing a backprojection,
An interpolation step for performing interpolation using a pixel value that is spatially and temporally adjacent to a missing pixel on the reconstructed image plane from which no pixel value is obtained by the backprojection step;
For each pixel on the reconstructed image satisfied by the backprojection step and the interpolation step, a spectrum estimation step for performing an original spectrum estimation for a wavelength using wavelength information of an optical filter corresponding to the image;
To obtain a spectral image of the subject.

A fifth invention made to achieve the above object is an image processing method for reconstructing a spectral image of a subject based on a plurality of single-eye images in the compound-eye imaging apparatus according to the first invention,
Assuming a reconstructed image plane obtained by adding optical filter type identification information and time information to each coordinate on a virtual object plane assuming a separation distance from the light receiving surface of the solid-state imaging device, a reconstructed image on the plane is assumed. From each of the above pixels, a projecting step of finding one or more single-eye image pixels that are assigned the same type of optical filter by geometric projection and that correspond spatially and temporally;
A pixel value of one pixel in the single-eye image found by the projection step is assigned to a pixel on the reconstructed image, or a pixel is obtained by correction processing using pixel values of corresponding pixels in the plurality of single-eye images. A pixel value determining step of calculating a value and applying the calculated value to a pixel on the reconstructed image;
For each pixel on the reconstructed image satisfied by the projection step and the pixel value determination step, a spectrum estimation step for performing an original spectrum estimation for a wavelength using wavelength information of an optical filter corresponding to the image;
To obtain a spectral image of the subject.

  Note that the compound eye imaging device used in the image processing method according to the fourth or fifth invention provides time information to each individual eye image by using, for example, an exposure / readout function by a rolling shutter of a solid-state imaging device. be able to.

  According to the image processing method of the fourth or fifth aspect of the invention, a comparatively high-resolution spectral image can be obtained by a comparatively simple arithmetic process.

A sixth aspect of the invention made to achieve the above object is an image processing method for reducing defocusing by subjecting each individual eye image to image processing in the compound eye imaging device according to the third aspect of the invention.
An area division processing step that divides a single-eye image into a central area and a plurality of annular areas surrounding the central area and removes blur by a single inverse filter for each divided area;
A region integration step of integrating the regions after processing by the region division processing step;
It is characterized by having.

  As described above, in the compound eye imaging device according to the third aspect of the present invention, the point spread function for the same image height is aligned in rotational symmetry, so that a point image is obtained in each of a plurality of regions into which a single eye image is divided. The change of the distribution function is small. Therefore, it is possible to remove the focal blur using a single inverse filter.

  In general, in order to obtain a deep depth of focus in a compound-eye imaging device, after estimating the distance from the device to the subject, a point spread function including blur corresponding to the distance is calculated, and complicated blurring calculation is performed to reduce the focal blur. It was necessary to remove. On the other hand, according to the image processing method of the sixth invention, since the point spread function depends only on the image height, it is deep with a small amount of calculation by simply combining a plurality of inverse filters without performing iterative calculation. The depth of field can be obtained.

  In the image processing method according to the sixth aspect of the invention, the area dividing process step sets each area so that a part thereof overlaps with the boundary between a plurality of adjacent areas, and the area integration step includes a part where the areas overlap. It is preferable to perform the joining process for. For example, α blending can be used for the joining process. According to this, since the image connection at the boundary of the region becomes good, the accuracy of distance measurement can be improved.

  According to the compound eye imaging device according to the first aspect of the invention, it is possible to appropriately extract the parallax information between the single-eye images without being substantially affected by the color of the subject, that is, the reflection spectrum distribution of the subject. Since the decrease in the amount of light is small, sensitivity is improved. Thereby, for example, the ranging accuracy of the subject can be improved. In addition, according to the image processing method according to the fourth and fifth inventions using the compound eye imaging device according to the first invention, with relatively simple calculation, that is, with hardware in a short time and relatively low cost, A moderately high resolution spectral image can be reconstructed.

  According to the compound eye imaging device of the second invention, the time intervals between corresponding pixels in a plurality of single-eye images are slightly non-uniform, so that the generation of zero points in the time-frequency component is avoided, and the high-definition image is accordingly generated. Can be reconfigured.

  In addition, according to the compound eye imaging device according to the third invention and the image processing method according to the sixth invention using the compound eye imaging device, it is possible to perform relatively simple calculation, that is, with a short time and relatively low cost hardware. A deep depth of field can be realized. Thereby, for example, the range of distance measurement can be expanded more than before without using mechanical autofocus or the like.

1 is a schematic configuration diagram of a main part of a compound eye imaging apparatus that is an embodiment of the present invention. Explanatory drawing of arrangement | positioning of the optical lens and optical filter in the compound-eye imaging device of a present Example. Explanatory drawing of arrangement | positioning of the optical lens in the compound-eye imaging device of a present Example. The figure which shows the concept of the information contained in the image signal obtained by one pixel on a light-receiving surface. The conceptual diagram which shows the flow of the process from the image input in the compound eye imaging device of a present Example to an image output. The conceptual diagram which shows the temporal relationship of each single eye image contained in a compound eye image. The conceptual diagram which shows the imaging model in the compound eye imaging device of a present Example. The conceptual diagram of the process of the 1st algorithm in a simple-type reconstruction method. The conceptual diagram which shows the correction method of the defect pixel in the algorithm of FIG. The conceptual diagram of the process of the 2nd algorithm in a simple reconstruction method. The flowchart which shows an example of the procedure of a basic ranging process. The conceptual diagram of the process of the 1st algorithm of a ranging method. The conceptual diagram of the process of the 2nd algorithm of a ranging method. The conceptual diagram of the process of the 3rd algorithm of a ranging method. Explanatory drawing of an example of the arrangement | positioning determination method of an optical lens and an optical filter. Explanatory drawing of the technique of the depth-of-field expansion in the compound-eye imaging device of a present Example. Explanatory drawing of the technique of the depth-of-field expansion in the compound-eye imaging device of a present Example. Explanatory drawing of the technique of the depth-of-field expansion in the compound-eye imaging device of a present Example. The schematic diagram for demonstrating the effect of having used the filter with correlation in the compound-eye imaging device of a present Example. The figure which shows the relationship between the evaluation value at the time of using a filter with correlation in the compound-eye imaging device of a present Example, and distance.

Hereinafter, a compound eye imaging apparatus according to an embodiment of the present invention will be described with reference to the accompanying drawings.
FIG. 1 is a schematic configuration diagram of a main part of the compound-eye imaging apparatus of the present embodiment, and FIGS. 2 and 3 are explanatory diagrams of the arrangement of optical lenses and optical filters in the compound-eye imaging apparatus of the present embodiment.

  As illustrated in FIG. 1, the compound-eye imaging apparatus according to the present embodiment includes an image input unit 1 that forms an image of light arriving from an object 100 and a plurality of single-eye images acquired by the image input unit 1. An A / D conversion unit 10 that digitizes an image signal corresponding to a compound eye image, a data processing unit 11 that generates a reconstructed image (spectral image) based on the digitized image signal, and a reconstructed image And a display unit 12 for displaying.

  The image input unit 1 includes a single solid-state imaging device 2 that is typically a CMOS image sensor, and a plurality (16 in FIG. 1) of micro optical lenses 5 that collect light coming from the subject 100, respectively. A lens array 4, a filter array 6 including a plurality of optical filters 7 disposed approximately one-on-one with the lens 5 immediately before each micro optical lens 5, and the light receiving surface 3 of the lens array 4 and the solid-state imaging device 2. And a partition wall member 8 provided between the two.

  One optical filter 7, one corresponding micro optical lens 5, and a divided region of the light receiving surface 3 on the solid-state imaging device 2 that captures a single-eye image formed by the micro optical lens 5; Is a single eye unit 9, the image input unit 1 can be regarded as being composed of a plurality (16 in FIG. 1) of individual eye units 9. Further, one optical filter 7 and one corresponding micro optical lens 5 can be regarded as one lens / filter unit. However, as will be described later, some of the micro optical lenses 5 may not be provided with the optical filter 7. The partition member 8 regulates the traveling direction and spread of the light so that the light passing through the micro optical lenses 5 and traveling toward the light receiving surface 3 of the solid-state imaging device 2 does not interfere between the individual unit 9. It is. In FIG. 1, the number of imaging units is 16, and in FIG. 2, the number is 25. However, the number of single-eye units 9 may be larger or may be smaller.

  As described in FIG. 1, the two-dimensional arrangement direction of the pixels arranged two-dimensionally on the light receiving surface 3 of the solid-state imaging device 2 is defined as x ′ direction and y ′ direction, and x ′ and y ′. The orthogonal height direction is z. The lens array 4 and the filter array 6 are two-dimensionally arranged in a plane substantially perpendicular to the z-axis, that is, in a plane substantially parallel to the light receiving surface 3 (however, between the lens array 4 and the light receiving surface 3). In this case, an arrangement in which the separation distance is slightly different for each micro optical lens 5 is also included. On the other hand, as shown in FIGS. 1 and 2A, the arrangement of the micro optical lenses 5 in the lens array 4 and the arrangement of the optical filters 7 in the filter array 6 are not along the x ′ axis and the y ′ axis. The lens / filter unit arrangement direction is shifted with respect to the pixel arrangement direction (x ′, y ′ direction) on the light receiving surface 3 so as to rotate by a predetermined angle Θ about the z axis. Accordingly, a state in which the micro optical lens 5, the optical filter 7, and the light receiving surface 3 of the solid-state imaging device 2 are viewed from the subject 100 in the z-axis direction is as shown in FIG. As will be described later, the y ′ direction is also a direction in which a rolling shutter is executed when a signal is read from the solid-state imaging device 2. Note that the two translation vectors that determine the arrangement of the micro optical lenses are not necessarily orthogonal, and the micro optical lenses may be arranged in a rhombus shape.

  Each of the plurality of optical filters 7 included in the filter array 6 is not a bandpass filter that transmits only light in a specific narrow wavelength band, but an optical filter having a predetermined transmission spectrum. In the example of FIG. 2A, six types of optical filters 7a to 7f having different transmission spectra are used. The optical filters 7a to 7f can be, for example, six types of magenta, cyan, yellow, pink, blue, and near infrared. An example of the transmission spectrum of the optical filters 7a to 7f in that case is shown in FIG.

  In the compound eye imaging device of the present embodiment, the characteristics of the transmission spectra of the plurality of types of optical filters 7a to 7f are that any two types of optical filters are determined to have a correlation with respect to wavelength. That is, in any two types of combinations of the six types of optical filters 7a to 7f, there is a common wavelength or wavelength band that can substantially obtain a certain degree of transmittance. Here, “substantially a certain degree of transmittance” is about 0.3 to 0.5 when viewed from the spectrum of FIG.

The filter array 6 includes approximately the same number of optical filters 7a to 7f. In addition, the positions of the optical filters of the same type are assigned such that the intervals are varied as much as possible in each of the row direction and the column direction (X ′ direction, Y ′ direction). For example a lens filter units 'm _x row direction, Y' X is a set of a micro-optical lens 5 and the optical filter 7 direction m _y columns (the total number: m _x × m _y) if located on the , '1 to m _x -1 in the direction, Y' X may different intervals between 1 to m _y -1 in the direction so as to mix. The "distance" of a lens filter unit referred to herein, as shown in FIG. 3, m _x rows, m _y columns, the center of the lens filter unit on the grid points of equal pitch intervals is placed flush in The distance between adjacent lattice points is assumed to be 1. Therefore, the “interval” of the lens / filter unit is an integer such as 1, 2, 3,.

However, the lens filter unit m _x rows, m _y column, rather than being disposed on the lattice points of a complete equal pitch interval, as shown in FIG. 3 the right, respectively lattice points in the same plane They are displaced from the top with a small amount of displacement. The minute displacement amount and the displacement direction are preferably different for each lens / filter unit. For example, the displacement amount and the displacement direction may be randomly given under a predetermined constraint condition (such as an upper limit of the displacement amount). . The advantages of adopting such a characteristic structure and arrangement will be described later.

Here, an example of a method for determining the arrangement of the optical lens and the optical filter will be described with reference to FIG.
First, the size of the lens array 4, that is, the number of micro optical lenses 5 arranged in the X ′ direction and the Y ′ direction (N _X × N _Y ) and the number of types of optical filters 7 (N _W ) are set. . For example, in the example of FIG. 15, N _X × N _Y = 5 × 5 and N _W is 6. Next, with respect to the two-dimensional lens arrangement (N _X × N _Y ), the number of optical filters 7 for each type to be arranged in the circumference is determined one round from the outer circumference side. Basically, the number of all types of optical filters 7 is made equal in one rotation. However, when the number of optical filters 7 cannot be equally divided, for example, an appropriate allocation such as increasing the number of important optical filters 7 is used. Adjustments shall be added. When the number of optical filters 7 to be assigned to one round is determined, the process moves to the inner circumference side, and the number of optical filters 7 to be assigned to one inner circumference side is determined. In this way, the number of assignments is sequentially determined up to the innermost circumference, but if there is a large (two or more in this case) non-uniformity in the number of assignments of each type of optical filter as a whole, the assignment is returned to the outer circumference side. Review.

  In the example of FIG. 15, as shown in FIG. 15A, the number of optical filters that can be arranged in one outermost circumference is 16, so three important four types of optical filters f1, f2, f3, and f4 are provided. Each of the other two types of optical filters f5 and f6 is assigned. The number of optical filters that can be arranged in one inner circle is eight. Since the optical filters f5, f6 are smaller in number than the optical filters f1, f2, f3, f4 on the outer peripheral side, two optical filters f5, f6 are assigned here, and the other four types of optical filters f1, f2 are assigned here. , F3, and f4 are assigned one by one. As a result, all of the six types of optical filters f1 to f6 are equally assigned by four. Here, an optical filter is not assigned to one central micro optical lens, and a single eye image obtained by this single eye unit is used as a reference.

When the assignment of a plurality of types of optical filters is completed as described above, the interval between the same type of optical filters is 1 to N _x −1 (here, in both the X ′ direction and the Y ′ direction, depending on the number of assigned numbers. Then, 4), the position of the optical filter is changed so as to vary as much as possible from 1 to N _Y −1 (here, 4). In practice, it is difficult to make the intervals uniform and uniform for all types of optical filters. Therefore, it is preferable to preferentially arrange important optical filters. In the example of FIG. 15, f1, f2, and f3 are preferentially arranged as particularly important optical filters so that the intervals are evenly distributed. As a result, the arrangement shown in FIG. 15B is obtained.
Of course, the above procedure is an example, and the arrangement determination method is not limited to this.

FIG. 5 is a conceptual diagram showing the flow of processing from image input to spectral image output in the compound eye imaging apparatus of the present embodiment.
When light arriving from the subject 100 is captured by the image input unit 1 described above, an image signal obtained by photoelectrically converting a plurality of single-eye images formed in parallel on the light receiving surface 3 of the solid-state imaging device 2 is input to the data processing unit 11. Entered. That is, an image signal corresponding to a compound eye image including the same number of single-eye images as the single-eye unit 9 is input to the data processing unit 11. The data processing unit 11 obtains a distance distribution indicating distance information for each pixel by performing distance measurement processing on the plurality of single-eye images, and uses the distance distribution to obtain a multidimensional point spread function. Then, using these pieces of information, either image reconstruction processing or super-resolution processing is executed to finally form a plurality of spectral images. Hereinafter, these processes will be described.

  First, a compound eye image obtained by the image input unit 1 will be described. FIG. 4 is a diagram showing a concept of information included in an image signal obtained by one pixel in FIG. As described above, since the plurality of single-eye units 9 are at different positions, the image signal includes information on the spaces x, y, and z indicating the positional relationship between the subject 100 and the single-eye unit 9. Further, since light arriving from the subject 100 passes through the optical filter 7 before reaching the light receiving surface 3 of the solid-state imaging device 2, information on the wavelength λ relating to the optical filter 7 (strictly speaking, solid-state imaging) (Including the wavelength characteristics of photoelectric conversion of the element 2). Furthermore, in this solid-state imaging device 2, the charge accumulation timing is not the same for all the pixels on the light receiving surface 3, and the rolling shutter is used to sequentially expose each line in the direction shown in FIG. The signal obtained by performing is read. Therefore, information on time t is also included in the image signal.

  FIG. 6 is a conceptual diagram showing a temporal relationship between individual eye images included in a compound eye image. As described above, since the pixel arrangement on the light receiving surface 3 of the solid-state imaging device 2 and the arrangement of the lens / filter unit are shifted by an angle Θ, on the light receiving surface 3, as shown in FIG. A single eye image is formed in a state shifted in the y ′ direction. As shown in FIG. 6B, exposure / signal readout by the rolling shutter is executed in order in the y ′ direction for each line in the x ′ direction. Therefore, for example, as shown in FIG. 6C, the nine single-eye images formed on the light receiving surface 3 of the solid-state imaging device 2 are regarded as continuous images that are slightly shifted in time in the order of numbers 1 to 9. Can do.

  When the lens and filter units are arranged in an orderly manner on the lattice points at equal pitch intervals shown in FIG. 3, the time shifts (T / N in FIG. 6B) of the individual eye images are the same. In that case, the time shift between the corresponding pixels on the single-eye image is the same regardless of the combination of any single-eye image that is temporally continuous. On the other hand, in the compound-eye imaging apparatus of this embodiment, since the minute displacement is independently given to each lens / filter unit as described above, the time shifts of the individual eye images are not completely the same. As a result, the time difference is slightly different depending on the combination of single-eye images that are temporally continuous. Since the time difference is slightly different in this way, reproducibility in the time direction is improved when image reconstruction is performed from a compound eye image using a super-resolution reconstruction method described later.

FIG. 7 is a conceptual diagram showing an imaging model in this compound eye imaging apparatus. As shown in FIG. 7A, an image point on the light receiving surface 3 for an individual unit i of an object point P (x, y, z) existing on a certain virtual object (subject) is represented by P ′ ( x ′, y ′, 0). When the information held by the subject, that is, the original information is f (x, y, z, λ, t), the compound eye image g (x ′, y ′) is defined by the following equation (1).
τ _{c (i)} (λ): Transmission spectrum of the optical filter type c included in the single eye unit i τ _p (x ′, y ′, λ): Captures the image point P ′ (x ′, y ′, 0) Transmission spectrum of optical filter corresponding to pixel s (λ): wavelength dependence of photoelectric conversion characteristics of solid-state imaging device 2 h _i (x, y, z, λ; x ′, y ′): included in individual eye unit i Imaging characteristics of the optical lens to be obtained w _t (y ′, t): accumulation time window width of the pixel at the position of y ′
(u _i , v _i ): center coordinates of the micro optical lens included in the single eye unit i f _i : focal length of the micro optical lens included in the single eye unit i (assuming that it is a pinhole lens)

The accumulation time window width w _t (y ′, t) is defined by the following equation.
Here, T _a is the accumulation time of the solid-state image sensor 2, and y _s and Δ _y ′ are related to the position and pitch of each pixel on the light receiving surface 3 of the solid-state image sensor 2 as shown in FIG. It is a parameter.
As can be seen from the equation (1), the compound eye image g (x ′, y ′) is originally the spatial information x associated with the imaging with respect to the original information f (x, y, z, λ, t) that the subject 100 has. , Y, z, wavelength information λ, and time information t are five-dimensional information superimposed. The operation for obtaining the original information f (x, y, z, λ, t) from the five-dimensional information g (x ′, y ′) is an image reconstruction process executed by the data processing unit 11 and is necessary at that time. It can be said that it is the distance measurement process that obtains such z information.

  Image reconstruction methods in the compound-eye imaging apparatus of the present embodiment are roughly classified into a simple reconstruction method and a super-resolution reconstruction method. Next, these methods will be described.

[Simple reconfiguration method]
The simple reconstruction method is literally a method of reconstructing a higher resolution image based on a low resolution compound eye image by relatively simple arithmetic processing. One of two algorithms can be adopted as the simple reconstruction method. In any algorithm, the imaging model described with reference to FIG. 7A is considered, and for a certain virtual object, the three-dimensional information called coordinates (x, y, z) of each object point on the virtual object is optically converted. Assume a five-dimensional reconstructed image to which two-dimensional information of filter type identification number c and accumulation start time t is added. An operation for filling each pixel on the reconstructed image is an image reconstruction process.

[1] First algorithm of simplified reconstruction method (see FIG. 8)
In the first algorithm, the pixel value of each pixel of the compound eye image corresponds to the type identification number c of the optical filter assigned to the monocular image in which the pixel exists and the accumulation start time t of the pixel signal. Back projection is performed on a virtual plane (a four-dimensional image of x, y, c, t) existing at a position separated by a predetermined distance z. For example, in the example of FIG. 8, the pixel value of a certain pixel (x ′, y ′) on the compound eye image is set as the type identification number c (in this example, c) of the optical filter 7 included in the unit eye unit 9 to which the pixel belongs. = # 6), the pixel coordinate (x, y) on the reconstructed image calculated from the pixel coordinate (x ′, y ′) on the compound eye image, and the accumulation start time t calculated from the coordinate y ′ (in this example) t = t _m-1 ) and a distance z given by the pixel coordinates (x, y) and the accumulation start time t. Each pixel on the reconstructed image is filled by back projecting the pixel values of all the pixels on the compound eye image onto the virtual plane in the same manner.

  However, in such back projection, pixel values are not always given to all coordinates on the virtual object, and defective pixels for which information is not given are generated. Therefore, as shown in FIG. 9, a plurality of defective pixels are surrounded by a plurality of pixels (four-dimensional surroundings including wavelength λ and time t, which are shown only in two dimensions x and y in FIG. 9). The pixel value is estimated by interpolation processing using the pixel value of the sufficient pixel, and is filled with the estimated pixel value. Thereby, the pixel values of all the pixels on the reconstructed image are determined. Thereafter, the original spectrum is estimated with respect to the wavelength of the pixel (x, y, z, t) on the reconstructed image using the information of the optical filter type identification number c. Thereby, a spectral image of the subject is acquired.

[2] Second algorithm of simplified reconstruction method (see FIG. 10)
In the second algorithm, contrary to the first algorithm, a plurality of corresponding pixels in the compound eye image are found from one pixel (x, y, z, c, t) on the reconstructed image, and the plurality of the pixels are found. A pixel on the reconstructed image by integrating the pixel value of the pixel, the optical filter type identification number c of the single eye image, and the accumulation start time t of the pixel signal, or by selecting one of the plurality of pixels Fill. For example, in the example of FIG. 10, for a certain pixel (x, y, c, t) in a four-dimensional space, for a single eye image having the optical filter type identification number c (c = # 6 in this example), Calculate the pixel coordinates (x ′, y ′) on the compound eye image calculated from the pixel coordinates (x, y) on the reconstructed image, and the accumulation start time t ′ calculated from the coordinates y ′, and start accumulation A _single eye image with the time t 'closest to t (in this case, t = t _m-1 ) is selected. Then, the pixel value of the pixel at the pixel coordinates (x ′, y ′) of the individual eye image is set as the pixel value of the pixel (x, y, c, t) on the original reconstructed image. Alternatively, instead of selecting a single-eye image whose accumulation start time t ′ is closest to t, a plurality of single-eye images that are close to t are selected, and a large weight is given in the closest order, and interpolation processing is performed from pixel values of a plurality of corresponding pixels. The pixel value may be obtained.

  By performing the same processing for the pixel values of all the pixels on the reconstructed image, all the pixels can be filled. In this case, since a pixel value is obtained for each pixel on the reconstructed image, a defective pixel as in the first algorithm does not occur, and interpolation processing for the defective pixel is unnecessary. Similar to the first algorithm, the original spectrum is estimated with respect to the wavelength using the information of the optical filter type identification number c for the pixel (x, y, z, t) on the reconstructed image. Thereby, a spectral image of the subject is acquired.

[2] Super-resolution reconstruction method In the super-resolution reconstruction method, a BTV (bilateral total variation) method conventionally known for creating a high-resolution image from a plurality of low-resolution images. Is used. Since this technique is not the gist of the present invention, a simple explanation will be given by giving an expression. In the super-resolution reconstruction method, among the five-dimensional functions shown in equation (1), a four-dimensional point spread function is obtained by adding two dimensions of wavelength and time to two dimensions in the spatial direction of x and y. Use. The basic equation for the BTV method is the following equation (2). X is a two-dimensional image of x, y represented by a column vector.
However, Argmin _X F (X) means X that minimizes F (X).

By adding two dimensions of wavelength and time to X, it is assumed that a four-dimensional image of x, y, λ, and t is represented by a column vector, and the above expression (2) is expanded to obtain expression (3).
Thereby, the bilateral constraint is applied not only in the four-dimensional point space direction but also in the wavelength and time directions.

Furthermore, in order to smooth the wavelength distribution, the m-dimensional wavelength information is converted into the wavelength information normalized by the luminance, or the luminance and the m−1-dimensional wavelength information, and the spatial smoothness of each wavelength component is constrained. Conditions and spatial smoothness constraint conditions are added to the combination of all two wavelength components. The basic equation of the BTV method when considering a color image is the following equation (4).

Extending this, the following equation (5) is obtained.
As is generally known, the BTV method has high robustness against noise, so that even when the S / N ratio of a compound eye image is not very good, it is possible to reconstruct a high resolution image without being influenced so much.

[Distance measurement method]
Next, a distance measuring method for estimating the distance of each object point on the virtual object from the compound eye image in the compound eye imaging apparatus of the present embodiment will be described. First, an example of a basic distance measurement procedure will be described with reference to FIG.

First, a search distance group Z = {z ₁ , z ₂ ,..., Z _nz }, which is a set of a plurality of search distances, is set. Since each search distance belonging to the search distance group Z is a distance candidate, for example, a predetermined distance range may be set with a predetermined step width without omission. Next, one unsearched search distance is selected from the search distance group Z, and this is set as a search trial distance z (step S2). Thereafter, a reconstructed image is created under the condition of the search trial distance z (step S3). For example, the simplified reconstruction method can be used.

  Then, one pixel (x, y) whose evaluation value is not calculated, which will be described later, on the generated reconstructed image is selected (step S4). A pixel value on another reconstructed image corresponding to or near the pixel (x, y) is acquired (step S5). Then, an evaluation value e is calculated (step S6). Thereafter, it is determined whether or not the evaluation value e has been calculated for the pixel (x, y) for the first time (step S7). If it is the first time, the evaluation value e is set to E (x, y) and the search trial is performed. The distance z is stored in D (x, y) (step S9). On the other hand, if it is determined in step S7 that the evaluation value e is not calculated for the first time, it is determined whether or not the evaluation value e is larger than the stored E (x, y) (step S8). If e> E (x, y), the process proceeds to step S9, where the stored E (x, y) is updated to e, and the search trial distance storage value D is also updated.

  If the evaluation value e is equal to or less than E (x, y) in step S8, and after the updating of E and D is completed in step S9, whether the evaluation value e has been calculated for all the pixels on the reconstructed image. (Step S10), and if there is a pixel whose evaluation value has not been calculated, the process returns to step S4. Therefore, by repeating steps S4 to S10, evaluation values for all pixels on the reconstructed image for a certain search trial distance z are calculated. If it determines with Yes at step S10, it will progress to step S11 and it will be determined whether the process with respect to all the search distances in the search distance group Z was implemented. If there is an unsearched distance, the process returns to step S2. Accordingly, by repeating steps S2 to S11, distance search based on the evaluation values for all the pixels on the reconstructed image is performed for all search distances set as the search distance group Z. Finally, the remaining D (x, y) is output as the estimated distance of each pixel (step S12).

Any of the following three algorithms can be adopted as the distance measuring method.
[1] First algorithm of distance measuring method (see FIG. 12)
In the first algorithm, one optical compound image is obtained from the compound eye image for the same type of optical filter (filter type: C _BASE ) as the reference single-eye image in the compound eye image under a condition assuming a certain distance z (or distance distribution). An image reconstructed from a compound eye image with respect to an optical filter (filter types: C ₁ , C ₂ , C ₃ , C ₄ , C ₅ , C ₆ ) different from the reference _single- eye image. Are converted into black and white (extracting only luminance information) and integrated to create two types of reconstructed images. The cross-correlation or mutual information amount of the two types of reconstructed images is calculated for each corresponding pixel, and the sum of the calculated values is obtained within a window having a predetermined size centered on each pixel, and this is used as an evaluation value. To do. The distance (or distance distribution) that maximizes the evaluation value is the distance of the pixel.

[2] Second algorithm for distance measurement method (see FIG. 13)
In the second algorithm, an optical filter (filter type: C _BASE ) of the same type as the reference single-eye image in the compound-eye image under a condition assuming a certain distance z (or distance distribution) The image is reconstructed, and the image from the compound eye image is obtained for each type of optical filter (filter types: C ₁ , C ₂ , C ₃ , C ₄ , C ₅ , C ₆ ) different from the reference single-eye image. Reconfigure. Then, each cross-correlation or mutual information amount of the former one reconstructed image and the same number of reconstructed images as the number of optical filter types is calculated for each pixel. Then, as in the first algorithm, the sum of the above calculated values is obtained within a window of a predetermined size centered on each pixel, and this is used as an evaluation value, and the distance (or distance distribution) that maximizes the calculated value is The pixel distance.

[3] Third algorithm of distance measuring method (see FIG. 14)
In the third algorithm, as in the first and second algorithms, without creating a reconstructed image, a plurality of pixels including a pixel (x, y) on the reconstructed image or pixels in the vicinity thereof are included. A set of corresponding pixel values on all _single- eye images having the same type of optical filter as the reference single-eye image (filter type: C _BASE ), and other optical filter types (filter types: C ₁ , C ₂ , C ₃ , C ₄ , C ₅ , C ₆ ), and a set of corresponding pixel values of a single eye image. Then, the sum of the mutual information between the pixel value group related to the reference single-eye image and the pixel value groups of the different optical filters is used as an evaluation value, and the distance or distance distribution that maximizes this is set as the pixel (x, y). Distance. Although this algorithm does not create a reconstructed image, it can be interpreted as performing substantially the same processing as that for creating a reconstructed image.

  In any of the distance measuring methods described above, the size of the window for calculating the pixel evaluation value is not fixed, but is adaptively changed so as to increase the distance measurement accuracy, or the weight of the pixel in the window considered in the evaluation value May be adaptively changed.

  As described above, the distance distribution of the entire subject can be obtained by calculating the distance for each pixel on the subject by combining any of the distance measuring methods and the image reconstruction method. Conversely, if the accuracy of the distance distribution increases, it can be used to increase the resolution of the reconstructed image.

  In particular, in the compound-eye imaging apparatus of the present embodiment, as described above, as the plurality of optical filters 7 included in the filter array 6, an optical filter having a correlation with respect to a wavelength is used instead of a bandpass filter, thereby improving the accuracy of distance measurement. Can be increased. The reason will be described with reference to the simplified schematic diagram shown in FIG. 19 and the relationship between the evaluation value and the distance shown in FIG. Now, it is assumed that the subject has a reflection spectrum biased as shown in FIG. That is, as shown in the right of FIG. 19A, three characters “A”, “B”, and “C” in which intensity peaks appear in isolation on the reflection spectrum are subjects.

  As shown in FIG. 19C, in the compound eye imaging device using the conventional bandpass filter, the transmission wavelength band is narrow. For this reason, on an image obtained by photographing a subject, characters having a reflection peak of approximately the same wavelength as the transmission peak of the bandpass filter are clearly projected, but characters having a reflection peak at a distant wavelength are thin. It is not projected or disappears at all. For this reason, the correlation between a plurality of images is low, and it is difficult to increase the accuracy of image matching. As a result, as shown in FIG. 20B, the evaluation value based on the plurality of reconstructed images becomes wider with respect to the search distance, and the peak height also decreases. For this reason, a false peak appearing at another distance greatly deviating from the true value is erroneously detected, and there is a high possibility that erroneous distance estimation is performed.

  On the other hand, as shown in FIG. 19B, in a compound eye imaging device using a filter having a high correlation with respect to the wavelength, three characters having different reflection peaks are almost clear on an image obtained by photographing a subject. It is projected on. For this reason, the correlation between a plurality of images is high, and alignment by image matching can be performed with high accuracy. As a result, as shown in FIG. 20A, the peak of the evaluation value with respect to the search distance appears clearly, and the distance closer to the true value can be estimated.

  The distance estimation method described above is based on the premise that there is no blur in the compound eye image. However, in reality, depending on the object distance, defocusing may occur, which may make distance estimation difficult. Then, next, the method employ | adopted in order to expand the depth of field in the compound-eye imaging device of a present Example is demonstrated with reference to FIGS. Here, a method called wavefront coding is used. However, in the original method, it is necessary to design the imaging optical system so that the point spread function does not depend on both object distance and image height. is there. This is a very strict requirement especially when the number of components of the lens is small as in a compound eye imaging apparatus. Thus, here, the point spread function is allowed to differ to some extent with respect to the image height, and this is compensated by image processing. That is, by intentionally giving each micro optical lens 5 included in the lens array 4 aberration such as spherical aberration, a point image is obtained at each image height within a certain object distance range regardless of the object distance. The distribution function is made almost constant in rotational symmetry with respect to the optical axis. At this time, the point spread function is allowed to differ with respect to the image height.

  Since the aberration is intentionally given, the single-eye image is totally blurred, but this out-of-focus blur is removed by the following image processing. As shown in FIG. 16A, with respect to a certain rectangular eye image, a central rectangular area <1> and a plurality of donut-shaped areas <2>, <3>, < 4> is divided into four areas. However, for example, the regions <1> and <2>, the regions <2> and <3>, and the regions <3> and <4> are set so as to partially overlap each other. As described above, since the point spread function is almost constant rotationally symmetrically with respect to the optical axis at each image height, the change in the point spread function can be regarded as small in each region. Therefore, blur is removed by applying a single inverse filter in each region.

  At this time, for the central region <1>, an inverse filter is applied to an image signal simply cut out from a single-eye image as shown in FIG. On the other hand, for the doughnut-shaped regions <2>, <3>, and <4>, in order to remove the influence of the rotational symmetry of the micro-optical lens, the xy-rθ transformation is performed to develop them into the rotating coordinate system. Then, an inverse filter is applied to the image signal included in each region after the conversion. Note that, when performing xy-rθ conversion in order to remove peripheral distortion (artifact) after the inverse filter processing, both ends of the θ overlap by an angle β.

As shown in FIG. 17 (a), with respect to the signal of each region after the inverse filter processing, a peripheral portion where an artifact may be generated is cut out, and then the regions <2>, <3>, and <4> are cut out. Returns to the donut-shaped region of the Cartesian coordinate system by performing the inverse rθ ”-xy conversion. When connecting the regions, α blending as shown in FIG. 18 is performed on the overlapping portion so that both sides of the boundary between the regions are smoothly connected.
FIG. 18 is a diagram showing the simplest one-dimensional α blending. When the image f _A (x) and the image f _B (x) are combined, the transmittance is determined so that α _A + α _B = 1 with respect to the overlapping portion, and the pixel value distribution f _A (x ) And the pixel value distribution f _B (x) in the region B, a composite image is created by the following equation.
g (x) = α _A f _A (x) + α _B f _B (x)

  As a result, a single-eye image from which the focal blur due to spherical aberration of the micro optical lens is removed is obtained, and a similar effect can be achieved over a certain wide object distance range.

  It should be noted that the above embodiment is merely an example of the present invention, and it is obvious that modifications, additions, and modifications as appropriate within the scope of the present invention are included in the scope of the claims of the present application. For example, in addition to using a wavelength filter as a micro optical filter, a combination of an ND filter, a polarizing filter, a phase plate, a diffraction grating, a hologram, and other various filters is also included in the modified form.

DESCRIPTION OF SYMBOLS 1 ... Image input part 2 ... Solid-state image sensor 3 ... Light-receiving surface 4 ... Lens array 5 ... Micro optical lens 6 ... Filter array 7, 7a-7f ... Optical filter 8 ... Partition member 9 ... Individual eye unit 10 ... A / D conversion Unit 11 Data processing unit 12 Display unit 100 Subject

Claims (9)

a lens array in which m (m is an integer of 4 or more) micro optical lenses are two-dimensionally arranged, and a single solid-state imaging device in which a plurality of micro light receiving units are two-dimensionally arranged. , A single-eye image of the subject by each of the micro optical lenses is formed in different areas on the light receiving surface of the solid-state imaging device, and image signals corresponding to m single-eye images are acquired, and a single image is obtained from this image signal. Alternatively, in a compound eye imaging device that generates a plurality of reconstructed images,
Optical filters having a predetermined wavelength transmission characteristic are arranged corresponding to all or a part of the m micro optical lenses, and the maximum m optical filters have different wavelength transmission characteristics n (n is 2). A compound-eye imaging device comprising at least one optical filter of the above integer) types, and any two of the n types of optical filters are correlated with respect to wavelength. The compound eye imaging device according to claim 1,
A compound-eye imaging apparatus, wherein the n types of optical filters are assigned to the m micro optical lenses with the same number or a maximum number difference of one. The compound eye imaging device according to claim 1,
The same type of optics so that the intervals between two adjacent optical filters of the same type are all the possible intervals in each dimension of the optical filters arranged two-dimensionally corresponding to the micro optical lens. A compound eye imaging device, wherein filters are arranged in a distributed manner. A lens array in which a plurality of micro optical lenses are two-dimensionally arranged, and one solid-state imaging device in which a plurality of micro light receiving units are two-dimensionally arranged, and each micro optical lens of the lens array The single-eye image of the subject imaged in step 1 is formed in different areas on the light receiving surface of the solid-state image sensor, and each single-eye image is sequentially read out with a time difference by a rolling shutter, and reconstructed from the read image signal In a compound eye imaging device that generates an image,
By disposing the plurality of micro optical lenses by shifting each micro optical lens independently from each other in a direction parallel to the two-dimensional plane from a position on a lattice point having an equal pitch in the row direction and the column direction, A compound-eye imaging apparatus characterized by giving different independent time differences to corresponding pixels of an image depending on a combination of single-eye images. A lens array in which a plurality of micro optical lenses are two-dimensionally arranged, and one solid-state imaging device in which a plurality of micro light receiving units are two-dimensionally arranged, and each micro optical lens of the lens array A single-eye image of the subject imaged in (1) is formed in different areas on the light receiving surface of the solid-state imaging device, and image signals corresponding to m single-eye images are obtained, and reconstructed images are obtained from the image signals. In a compound eye imaging device that generates
By giving aberration to each micro optical lens of the lens array, the distance between the device and the subject is within a predetermined range, and the point spread function for the same image height is rotated with respect to the optical axis regardless of the distance. A compound-eye imaging device characterized by being arranged symmetrically. An image processing method for reconstructing a spectral image of a subject based on a plurality of single-eye images in the compound-eye imaging device according to claim 1,
Assuming a reconstructed image plane obtained by adding the type identification information and time information of the optical filter to each coordinate on the virtual object plane assuming the separation distance from the light receiving surface of the solid-state image sensor, each pixel of each eye image The pixel value is geometrically plotted on the reconstructed image plane based on the pixel coordinates of the pixel on the light receiving surface, the type of optical filter assigned to the single eye image, and the accumulation start time at the solid-state imaging device at the pixel. A backprojecting step for performing a backprojection,
An interpolation step for performing interpolation using a pixel value that is spatially and temporally adjacent to a missing pixel on the reconstructed image plane from which no pixel value is obtained by the backprojection step;
For each pixel on the reconstructed image satisfied by the backprojection step and the interpolation step, a spectrum estimation step for performing an original spectrum estimation for a wavelength using wavelength information of an optical filter corresponding to the image;
An image processing method characterized in that a spectral image of a subject is acquired by executing. An image processing method for reconstructing a spectral image of a subject based on a plurality of single-eye images in the compound-eye imaging device according to claim 1,
Assuming a reconstructed image plane obtained by adding optical filter type identification information and time information to each coordinate on a virtual object plane assuming a separation distance from the light receiving surface of the solid-state imaging device, a reconstructed image on the plane is assumed. From each of the above pixels, a projecting step of finding one or more single-eye image pixels that are assigned the same type of optical filter by geometric projection and that correspond spatially and temporally;
A pixel value of one pixel in the single-eye image found by the projection step is assigned to a pixel on the reconstructed image, or a pixel is obtained by correction processing using pixel values of corresponding pixels in the plurality of single-eye images. A pixel value determining step of calculating a value and applying the calculated value to a pixel on the reconstructed image;
For each pixel on the reconstructed image satisfied by the projection step and the pixel value determination step, a spectrum estimation step for performing an original spectrum estimation for a wavelength using wavelength information of an optical filter corresponding to the image;
An image processing method characterized in that a spectral image of a subject is acquired by executing. An image processing method for reducing out-of-focus blur by performing image processing on each single-eye image in the compound-eye imaging device according to claim 3,
An area division processing step that divides a single-eye image into a central area and a plurality of annular areas surrounding the central area, and removes blur by a single inverse filter for each divided area;
A region integration step of integrating the regions after processing by the region division processing step;
An image processing method comprising: The image processing method according to claim 8, comprising:
The region division processing step sets each region so that a part thereof overlaps across a boundary between a plurality of adjacent regions, and the region integration step performs a joining process on a part where the regions overlap. Image processing method.