JP2012129654A - Imaging device, and control method and program therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging device, and control method and program therefor, capable of forming a color reconstruction image with high resolution, using imaging data acquired so as to include optical information in the traveling direction.SOLUTION: The light which has passed through an imaging lens is dispersed into at least two types of light with different spectral characteristics. The imaging device includes a micro lens array which corresponds to at least one imaging means of a plurality of imaging sections for receiving each of the two or more types of dispersed light to form imaging data.

Description

  The present invention relates to an imaging apparatus that captures an image, a control method thereof, and a program.

  An imaging device using a technique called “Light Field Photography” has been proposed (for example, Patent Document 1 and Non-Patent Document 1). This imaging device uses a microlens array composed of a plurality of microlenses, and in addition to the intensity distribution of light on the light receiving surface of the imaging element, can also obtain information on the traveling direction of the light. . After imaging, an image set at an arbitrary viewpoint or an arbitrary focus can be reconstructed using the acquired light intensity distribution and information on the traveling direction. Such an imaging device is called a light field camera in the present invention.

JP 2009-124213 A

Ren.Ng and 7 others, “Light Field Photography with a Hand-Held Plenoptic Camera”, Stanford Tech Report CTSR 2005-02

  However, in the light field camera, the information on the light traveling direction can be obtained, but the resolution of the reconstructed image is lowered. Further, in the colorization using the color filter proposed in Non-Patent Document 1 and Patent Document 1, each pixel of the image sensor corresponds to one of the color components, so that the resolution of each color component further decreases. End up.

  The present invention has been made in view of the above problems, and an imaging apparatus capable of generating a high-resolution color reconstructed image using imaging data acquired so as to include information on the light traveling direction, and the imaging apparatus It is an object to provide a control method and a program.

In order to achieve the above object, an imaging apparatus according to the present invention comprises the following arrangement. That is,
An imaging device that captures an image,
An imaging lens;
A spectroscopic means for splitting light that has passed through the imaging lens into at least two lights having different spectral characteristics;
A plurality of imaging means that respectively receive two or more lights separated by the spectroscopic means and generate imaging data;
A microlens array corresponding to at least one of the plurality of imaging means.

  According to the present invention, by using a light field camera having a plurality of image sensors, a color reconstructed image can be obtained without reducing the resolution of each color component. Furthermore, a higher-resolution color reconstructed image can be obtained by changing the presence or absence of the microlens array and the number of microlenses constituting the microlens array in accordance with the magnitude of the visual influence of each color component.

1 is a block diagram illustrating an overall configuration of an imaging apparatus according to Embodiment 1. FIG. FIG. 3 is a block diagram illustrating a functional configuration of a signal processing unit according to the first embodiment. FIG. 5 is a diagram schematically illustrating the operation of the color separation prism according to the first embodiment. It is a figure which shows typically the process of obtaining the imaging data of Embodiment 1. FIG. It is a figure which shows typically the information of the light ray contained in the imaging data of Embodiment 1. FIG. FIG. 3 is a diagram schematically illustrating a correspondence relationship between a pixel group of the image sensor of Embodiment 1 and a microlens. 3 is a flowchart illustrating an operation of a viewpoint information interpolation unit according to the first embodiment. FIG. 6 is a diagram schematically illustrating an operation of a viewpoint information interpolation unit according to the first embodiment. FIG. 6 is a diagram schematically illustrating an operation of a parallax image generation unit according to the first embodiment. 6 is a flowchart illustrating an operation of a parallax image interpolation unit according to the first embodiment. FIG. 6 is a diagram schematically illustrating the operation of the rearrangement processing unit according to the first embodiment. FIG. 6 is a diagram schematically illustrating an operation of an image reconstruction unit according to the first embodiment. FIG. 3 is a diagram schematically illustrating an optimal microlens arrangement according to the first embodiment. FIG. 6 is a diagram schematically illustrating a refocus calculation processing process of the first embodiment. FIG. 6 is a diagram schematically illustrating a correspondence relationship between a pixel group of an image sensor of Embodiment 2 and a microlens. 6 is a block diagram illustrating a functional configuration of a signal processing unit according to Embodiment 2. FIG.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
<Embodiment 1>
FIG. 1 is a block diagram illustrating the overall configuration of the imaging apparatus according to the first embodiment.

  The imaging apparatus includes an imaging lens 101, a color separation prism 102, and a microlens array 103 (a red (R) component microlens array 103R, a green (G) component microlens array 103G, and a blue (B) component microlens array. 103B). The imaging device includes an imaging element 104 (an R-component imaging element 104R, a G-component imaging element 104G, and a B-component imaging element 104B), an A / D conversion unit 105, a signal processing unit 106, an encoder unit 107, and a medium. I / F 108 is provided. Further, the imaging apparatus includes a CPU 109, a ROM 110, and a RAM 111. Various components of the image pickup apparatus such as an aperture and a shutter are present in addition to those shown in the figure, but are not the main points of the present invention, and thus description thereof is omitted.

  The imaging lens 101 collects object light from the imaging target. A color separation prism 102 serving as a spectroscopic unit splits light that has passed through the imaging lens 101 into at least two light beams having different spectral characteristics. Here, the color separation prism 102 separates the object light collected by the imaging lens 101 into RGB. The object lights separated by the color separation prism 102 are incident on the microlens arrays 103R, 103G, and 103B arranged on the image forming surface of the imaging lens 101, respectively. Each microlens array is composed of a plurality of microlenses.

  The imaging elements 104R, 104G, and 104B, which are imaging units, receive object light from the microlens arrays 103R, 103G, and 103B, respectively, and generate imaging data. Here, the image sensors 104R, 104G, and 104B are arranged on the image planes of the microlenses that constitute the microlens arrays 103R, 103G, and 103B, respectively. For example, a CCD (charge-coupled device) or a CMOS (Complementary Metal Oxide Semiconductor) sensor can be used as the image sensor 104. One microlens in the microlens array 103 is assigned to each of the plurality of pixels of the image sensor 104.

  Here, the number of microlenses composing one microlens array is the resolution of the reconstructed image described later. On the other hand, the number of pixels of the image sensor corresponding to one microlens corresponds to the number of parallaxes. As the number of parallaxes increases, the resolution of a reconstructed image, which will be described later, for example, the resolution at an arbitrary viewpoint, the resolution in the depth direction based on the refocus calculation process, or the like increases. The resolution and resolution of the reconstructed image are in a trade-off relationship.

  In the imaging device according to Embodiment 1, the number of microlenses constituting the microlens array is at least two or more. Specifically, the number of microlenses constituting the microlens array 103G is larger than the number of microlenses constituting the microlens arrays 103R and 103B. Thus, by relatively increasing the number of microlenses constituting the microlens array 103G, the resolution of the G component can be relatively increased. When the resolution of the R component and the B component is increased by interpolation processing to increase the resolution, the visual influence degree of the G component is large, so that a color reconstructed image with better resolution can be obtained. On the other hand, since the number of pixels of the R component and B component imaging elements corresponding to one microlens is large, the resolution of the R component and B component of the reconstructed image is not sacrificed.

  The A / D conversion unit 105 converts the amount of light received by the image sensor 104 into a digital value. The signal processing unit 106 performs predetermined signal processing (described later) on the digital value, and outputs a reconstructed image (described later). Details of the signal processing unit 106 will be described later.

  The encoder unit 107 performs processing for converting the reconstructed image into a file format such as JPEG. The media I / F 108 is an interface for connecting to a PC or other media (for example, hard disk, memory card, CF card, SD card, USB memory).

  The CPU 109 is involved in all the processes of each configuration, reads the instructions stored in the ROM 110 and the RAM 111 in order, interprets them, and executes the processes according to the results. The ROM 110 and RAM 111 provide the CPU 109 with programs, data, work areas, and the like necessary for the processing.

  FIG. 2 is a block diagram illustrating a functional configuration of the signal processing unit 106 according to the first embodiment.

  The signal processing unit 106 includes a viewpoint information interpolation unit 201, a parallax image generation unit 202, a parallax image interpolation unit 203, a rearrangement processing unit 204, a color imaging data synthesis unit 205, and an image reconstruction unit 206.

  The viewpoint information interpolation unit 201 generates imaging data obtained by performing interpolation processing for each pixel region corresponding to each microlens with respect to imaging data (G imaging data) obtained by the imaging element 104G. Details of the viewpoint information interpolation unit 201 will be described later.

  The parallax image generation unit 202 is arranged at the same position between the pixel areas corresponding to each microlens with respect to the imaging data (R imaging data and B imaging data) obtained by the imaging elements 104R and 104B. The extracted pixel data is synthesized. Here, the generated imaging data is an array of images obtained by imaging imaging objects from different viewpoints. Details of the parallax image generation unit 202 will be described later.

  The parallax image interpolation unit 203 generates imaging data obtained by performing interpolation processing for each pixel region corresponding to each viewpoint on the imaging data generated by the parallax image generation unit 202. Details of the parallax image interpolation unit 203 will be described later.

  The rearrangement processing unit 204 extracts and synthesizes pixel data arranged at the same position between the pixel regions corresponding to each viewpoint with respect to the imaging data generated by the parallax image interpolation unit 203. Here, the generated imaging data is obtained by arranging the imaging data corresponding to each microlens, like the imaging data obtained by the imaging element 104. Details of the rearrangement processing unit 204 will be described later.

  The color imaging data combining unit 205 generates color imaging data by combining the imaging data obtained by the viewpoint information interpolation unit 201 and the imaging data obtained by the rearrangement processing unit 204.

  The image reconstruction unit 206 generates an arbitrary viewpoint image and an arbitrary focus image from the color imaging data obtained by the color imaging data synthesis unit 205. Details of the image reconstruction unit 206 will be described later.

  In addition, the signal processing unit 106 may include other processing units such as noise reduction and edge enhancement.

  With reference to FIGS. 3 to 11, the operation of the imaging apparatus according to the first embodiment will be described.

  FIG. 3 is a diagram schematically illustrating how the object light is dispersed by the color separation prism 102 according to the first embodiment.

  The color separation prism 102 includes three prism blocks 301 to 303, prism blocks 301 and 302, and two dichroic mirrors 304 and 305 disposed on the joint surfaces of the prism blocks 302 and 303, respectively.

  The dichroic mirror 304 reflects the B component object light and transmits the R component and G component object light out of the light (indicated by the alternate long and short dash line) incident on the incident surface 306 of the prism block 301. The dichroic mirror 305 reflects the R component object light and transmits the G component object light out of the R component and G component object light transmitted through the dichroic mirror 304 and the prism block 302. The prism blocks 301 and 302 guide the B component object light and the R component object light reflected by the dichroic mirrors 304 and 305 to the exit surfaces 307 and 308, respectively. The prism block 303 guides the G component object light transmitted through the dichroic mirror 305 to the emission surface 309.

  FIG. 4 is a diagram schematically illustrating a process in which the imaging element 104 according to the first embodiment obtains imaging data of an imaging target.

  Object light from the imaging object 3010 is collected by the imaging lens 101. The light condensed by the imaging lens 101 is incident on the microlens array 103 disposed on the imaging surface of the imaging lens 101 (f1 in FIG. 4 is the focal length of the imaging lens 101). The light that has passed through the microlens array 103 reaches the image sensor 104 disposed on the image plane of the microlens that constitutes the microlens array 103 (f2 in FIG. 4 is the focal length of the microlens). Here, light rays that have passed through different locations of the imaging lens 101 reach different locations on the image sensor 104.

  FIG. 5 is a diagram schematically illustrating light ray information included in image data obtained by the image sensor 104 according to the first embodiment.

Consider an orthogonal coordinate system (u, v) on the imaging lens surface of the imaging lens 101 and an orthogonal coordinate system (x, y) on the imaging surface of the imaging element 104, respectively. When the distance between the imaging lens surface of the imaging lens 101 and the imaging surface of the imaging element 104 is F, the light rays passing through the imaging lens 101 and the imaging element 104 are represented by a four-dimensional function L _F (x, y, u, v). The L _F (x, y, u, v) includes information on the traveling direction of the light beam in addition to the position information of the light beam.

  FIG. 6 is a diagram schematically illustrating a correspondence relationship between a pixel group in the image sensor 104 of the first embodiment and a microlens constituting the microlens array 103.

  A square indicated by a solid line in FIG. 6 represents the imaging element 104, and a dotted line represents a microlens constituting the microlens array 103. In FIG. 6, the number of pixels of the image sensors 104R, 104G, and 104B is 18 × 18, one microlens corresponds to 3 × 3 pixels of the image sensor 104G, and 9 × of the image sensors 104R and 104B. One micro lens corresponds to 9 pixels.

  In the example of FIG. 6, the resolution of the G component of the reconstructed image described later is 6 × 6, the number of parallaxes is 3 × 3, the resolution of the R component and the B component is 2 × 2, and the number of parallaxes is 9 × 9. . Here, the G component has a relatively small number of parallaxes, while the R component and the B component have a relatively small resolution. The viewpoint information interpolation unit 201 performs an interpolation process so that the number of viewpoints of the G component is equal to the number of parallaxes of the R component and the B component. In the parallax image interpolation unit 203, the resolution of the R component and the B component is the resolution of the G component. Interpolation processing is performed so that Here, the parallax image generation unit 202 is preprocessing of the parallax image interpolation unit 203, and changes the arrangement of the imaging data in order to interpolate the resolution. The rearrangement processing unit 204 is post-processing of the parallax image interpolation unit 203, and returns the arrangement changed by the parallax image generation unit 202 to an arrangement in which regions corresponding to the microlenses are arranged.

  Note that the correspondence relationship between the pixel group and the microlens in FIG. 6 is an example, and the number of pixels of the imaging elements 104R, 104G, and 104B and the number of pixels of the imaging element corresponding to one microlens may be any combination. In addition, the arrangement of the microlenses is not limited to a matrix, and other arrangements such as a honeycomb may be used.

  FIG. 7 is a flowchart for explaining processing of the viewpoint information interpolation unit 201 according to the first embodiment. FIG. 8 is a diagram schematically illustrating processing of the viewpoint information interpolation unit 201 according to the first embodiment.

  In FIG. 8, imaging data 801 represents imaging data obtained by the imaging element 104G. The viewpoint information interpolation unit 201 performs an interpolation process on a region (for example, the region 802) corresponding to each microlens in the imaging data 801, thereby changing the number of parallaxes of the G component and the number of parallaxes of the R component and the B component. Make equal.

  In step S701, the viewpoint information interpolation unit 201 acquires imaging data obtained from the imaging element 104G. In step S702, the viewpoint information interpolation unit 201 divides the acquired imaging data into regions corresponding to each microlens. In step S703, the viewpoint information interpolation unit 201 sets the first area among the divided areas as the attention area. For example, the region 802 can be selected as the first region.

  In step S704, the viewpoint information interpolation unit 201 performs an enlargement process by interpolation so that the number of pixels in the attention area is the same as the number of pixels in the area corresponding to one microlens of the imaging elements 104R and 104B. For example, the interpolation process is performed so that the region 802 becomes the region 803. For this interpolation processing, for example, a bilinear method or a bicubic method can be used.

  In step S705, the viewpoint information interpolation unit 201 determines whether the interpolation processing for all regions has been completed. If the interpolation processing for all areas has not been completed (NO in step S705), the process proceeds to step S706. In step S706, the viewpoint information interpolation unit 201 sets the next region as a region of interest, and repeats the processing from step S704 to step S705. As a method for selecting the next region, for example, raster scanning depicted by arrows and dotted lines in FIG. 8 can be used. When the interpolation process for all regions is completed (YES in step S705), the interpolation process is terminated.

  Note that the method of selecting the first region and the next region is an example, and any order may be used as long as all regions can be interpolated.

  FIG. 9 is a diagram schematically illustrating processing of the parallax image generation unit 202 according to the first embodiment.

  The imaging data 901 represents imaging data obtained by the imaging elements 104R and 104B. The parallax image generation unit 202 extracts pixel data (for example, the pixels 901a to 901d and the pixels 901e to 901h of the imaging data 901) arranged at the same position between the pixel regions corresponding to each microlens. These are combined and arranged so as to correspond to the positions in the pixel area. The imaging data 902 represents imaging data obtained by rearranging the imaging data 901 by the parallax image generation unit 202, and the pixels 902a to 902h correspond to the pixels 901a to 901h, respectively.

  Here, the imaging data generated by the parallax image generation unit 202 is an array of images obtained by imaging imaging objects from different viewpoints. The parallax image interpolation unit 203 performs an interpolation process on the imaging data from each viewpoint. That is, the parallax image generation unit 202 performs preprocessing for performing interpolation processing on the imaging data from each viewpoint in the parallax image interpolation unit 203.

  FIG. 10 is a flowchart for explaining the processing of the parallax image interpolation unit 203 of the first embodiment.

  In step S <b> 1001, the parallax image interpolation unit 203 acquires imaging data (parallax image data) generated by the parallax image generation unit 202. In step S1002, the parallax image interpolation unit 203 divides the acquired imaging data into regions corresponding to the respective viewpoints. Here, for example, the imaging data 902a to 902d and 902e to 902h are one area. In step S1003, the parallax image interpolation unit 203 sets the first area among the divided areas as the attention area. As the first region, for example, as in the viewpoint information interpolation unit 201, the uppermost region can be selected.

  In step S1004, the parallax image interpolation unit 203 performs enlargement processing by interpolation so that the number of pixels in the region of interest is the same as the number of microlenses constituting the microlens array 103G. For this interpolation processing, for example, a bilinear method or a bicubic method can be used. Since the number of pixels in one area is the resolution of a reconstructed image, which will be described later, the interpolation processing in the parallax image interpolation unit 203 is equivalent to interpolating the resolutions of the R component and the B component.

  In step S1005, the parallax image interpolation unit 203 determines whether the interpolation processing for all regions has been completed. If the interpolation processing for all the areas has not been completed (NO in step S1005), the process proceeds to step S1006. In step S1006, the parallax image interpolation unit 203 sets the next region as a region of interest, and repeats the processing from step S1004 to step S1005. As a method for selecting the next region, for example, raster scanning can be used as in the viewpoint information interpolation unit 201. When the interpolation process for all regions is completed (YES in step S1005), the parallax image interpolation unit 203 ends the interpolation process.

  Note that the method of selecting the first region and the next region is an example, and any order may be used as long as all regions can be interpolated.

  FIG. 11 is a diagram schematically illustrating processing of the rearrangement processing unit 204 according to the first embodiment.

  The imaging data 1101 represents a part of imaging data generated by the parallax image interpolation unit 203. The rearrangement processing unit 204 extracts and synthesizes pixel data (for example, the pixels 1101a to 1101i in the imaging data 1101) arranged at the same position between the pixel areas corresponding to the respective viewpoints. To align with the position in the The imaging data 1102 represents imaging data obtained by rearranging the imaging data 1101 by the rearrangement processing unit 204. For example, the pixels 1101a to 1101i and the pixels 1102a to 1102i correspond to each other.

  The imaging data rearranged by the parallax image generation unit 202 is rearranged by the rearrangement processing unit 204. Therefore, the imaging data generated by the rearrangement processing unit 204 is an array in which pixel regions corresponding to each microlens are arranged corresponding to the position of the microlens, similarly to the imaging data generated by the viewpoint information interpolation unit 201. .

  That is, the process in the rearrangement processing unit 204 is a post-process for returning the array changed by the process of the parallax image generation unit 202, which is a pre-process for performing the interpolation in the parallax image interpolation unit 203. Here, the number of pixel areas corresponding to each microlens and the pixels constituting one pixel area, with the imaging data generated by the rearrangement processing unit 204 and the imaging data generated by the viewpoint information interpolation unit 201 Interpolation processing is performed so that the numbers are the same. The imaging data generated by the viewpoint information interpolation unit 201 and the imaging data generated by the rearrangement processing unit 204 are color-synthesized by the color imaging data synthesis unit 205.

  FIG. 12 is a diagram for explaining arbitrary viewpoint image generation in the image reconstruction unit 206 according to the first embodiment. FIG. 13 is a diagram schematically illustrating an optimal microlens arrangement for generating an arbitrary viewpoint image according to the first embodiment.

  In arbitrary viewpoint image generation, pixel data at the same position is extracted and synthesized between pixel regions corresponding to each microlens, thereby reconstructing an image set at an arbitrary viewpoint. For example, when pixel data arranged at the position of the pixel 1201a between pixel regions corresponding to each microlens is extracted and combined, an image from the viewpoint in the upper left direction is reconstructed. Similarly, if the pixel 1201b is used, The image from the viewpoint in the lower right direction is reconstructed.

  Here, since the G component of the image data obtained by the color image data synthesizing unit 205 interpolates the viewpoint information, there is a possibility that an error occurs in the luminance of the arbitrary viewpoint image or the color balance is lost. However, only the information that has not been interpolated is used by setting the number of pixels of the image sensors 104R and 104B corresponding to one microlens to an odd multiple of the number of pixels of the image sensor 104G corresponding to one microlens. Thus, it is possible to generate an image from a specific viewpoint.

  In FIG. 13, for example, the imaging data 1301 and 1302 includes the case where the number of pixels of the imaging element 104G corresponding to one microlens is set to 3 × 3 and the number of pixels of the imaging elements 104R and 104B to 9 × 9, respectively. The imaging data when set is shown. Further, the imaging data 1303 is obtained by performing interpolation processing on the imaging data 1301 by the viewpoint information interpolation unit 201. In the imaging data 1301, 1302, and 1303, pixels assigned the same number are light ray information from the same direction. For this reason, when viewpoint images from viewpoints assigned the same numbers in 1302 and 1303 are generated, interpolation is not performed, and thus luminance errors and color balance collapse due to interpolation do not occur.

  FIG. 14 is a diagram schematically illustrating a refocus calculation process for generating an arbitrary focus image in the image reconstruction unit 206 according to the first embodiment.

As shown in FIG. 14, the positional relationship among the imaging lens surface 1401, the imaging element surface 1402, and the refocus surface 1403 is set so that F ′ = αF. In this case, the detected intensity L _{F ′ at} the coordinates (s, t) on the refocus plane 1403 is expressed by the following equation (1).

Further, the image E _{F ′} (s, t) obtained on the refocus surface 1403 is obtained by integrating the detected intensity L _{F ′} by the area of the lens aperture, and is expressed as the following equation (2). The An arbitrary focus image can be reconstructed by performing the refocus calculation process from the equation (2).

  In the first embodiment, an example in which object light is decomposed into three components of RGB has been described. However, the object light is decomposed into four components of yellow (Y), magenta (M), cyan (C), and green (G). It doesn't matter. In that case, the number of microlenses constituting the microlens array for the G component is made larger than the number of microlenses constituting the microlens arrays for the Y component and the C component for the M component.

  In the first embodiment, the number of microlenses constituting the microlens arrays for the R component and the B component is the same, but may be different. In this case, interpolation is performed so that both the resolutions of the R component and B component imaging data are the same as the resolution of the G component imaging data.

  Furthermore, in the first embodiment, the resolution of the G component is relatively increased by increasing the number of microlenses constituting the microlens array for the G component, but the microlens array for the G component is disposed. By not doing so, the resolution of the G component may be relatively increased. In that case, the G-component image sensor is arranged on the imaging plane of the imaging lens. With such an arrangement, the same imaging data as when one microlens is associated with each pixel of the G component imaging element can be obtained.

  Furthermore, in the first embodiment, the image sensor is arranged for each color component. However, the dispersed object light may be acquired in color and monochrome, respectively. In that case, by disposing only the microlens array corresponding to monochrome, it is possible to estimate the distance of the captured object from monochrome imaging data. On the other hand, by disposing the microlens array corresponding to the color, a color image having the same resolution as that of a normal camera can be obtained. A high-resolution reconstructed image is obtained by performing signal processing on the color image according to the distance estimated from the monochrome imaging data.

  As described above, according to the first embodiment, the resolution of the G component is relatively increased by increasing the number of microlenses constituting the microlens array in the G component rather than the R component and the B component. Can do. Further, when the resolution of the R component and the B component is increased by interpolation processing to increase the resolution, the visual influence degree of the G component is large, so that a color reconstructed image with better resolution can be obtained. On the other hand, since the number of pixels of the R component and B component imaging elements corresponding to one microlens is large, there is an effect that the resolution of the R component and B component of the reconstructed image is not sacrificed.

<Embodiment 2>
In the first embodiment, the presence or absence of the microlens array and the number of microlenses constituting the microlens array are determined so that the resolution of the G component is higher than the resolution of the R component and the B component. In the second embodiment, a configuration for determining the presence / absence of a microlens array and the number of microlenses constituting the microlens array so that the resolution of the R component and the B component is higher than the resolution of the G component will be described.

  By increasing the number of microlenses constituting the microlens array 103 by the R component and the B component rather than the G component, the number of parallaxes of the G component can be relatively increased. By doing so, the G component does not need to interpolate the number of parallaxes. Since the G component has a large degree of visual influence, it is difficult to perceive the loss of color balance by not interpolating the number of parallaxes of the G component, and when the resolution of the G component is compensated by interpolation processing to increase the color balance, It is possible to obtain a high-resolution color reconstructed image that hardly perceives the collapse of the image.

  FIG. 15 is a diagram schematically illustrating a correspondence relationship between a pixel group in the image sensor 104 of the second embodiment and a microlens constituting the microlens array 103.

  In the second embodiment, contrary to the first embodiment, the number of pixels of the image sensor 104 corresponding to one microlens is larger in the R component and the B component than in the G component. That is, the resolution of the R component and the B component is higher than that of the G component, and the number of parallaxes of the G component is larger than the number of parallaxes of the R component and the B component.

  FIG. 16 is a block diagram illustrating a functional configuration of the signal processing unit 106 according to the second embodiment.

  In the second embodiment, contrary to the first embodiment, imaging data (G imaging data) obtained by the imaging element 104G is input to the viewpoint information interpolation unit 201. In addition, imaging data (R imaging data and B imaging data) obtained by the imaging elements 104 </ b> R and 104 </ b> B are input to the parallax image generation unit 202. In the second embodiment, the resolution of the G component is lower than that of the R component and the B component, and the number of parallaxes of the R component and the B component is lower than the number of parallaxes of the G component. Is the opposite.

  By performing other processing in the same manner as in the first embodiment, a color reconstructed image with high resolution can be obtained. At this time, since the interpolation processing is not performed on the number of parallaxes of the G component, a reconstructed image in which color balance is not easily lost can be obtained. Other processes are the same as those in the first embodiment, and thus the description thereof is omitted.

  As described above, according to the second embodiment, the number of parallaxes of the G component is relatively increased by increasing the number of microlenses constituting the microlens array in the R component and the B component rather than the G component. be able to. By doing so, the G component does not need to interpolate the number of parallaxes. Since the G component has a large degree of visual influence, a color reconstructed image in which color balance is not easily lost can be obtained by not interpolating the number of parallaxes of the G component. Further, when the resolution of the G component is increased by interpolation processing to increase the resolution, there is an effect of obtaining a high-resolution color reconstructed image that hardly perceives color balance collapse.

  The present invention can also be realized by executing the following processing. That is, software (program) for realizing the functions of the embodiment is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads out and executes the program. It is a process to control.

Claims (10)

An imaging device that captures an image,
An imaging lens;
A spectroscopic means for splitting light that has passed through the imaging lens into at least two lights having different spectral characteristics;
A plurality of imaging means that respectively receive two or more lights separated by the spectroscopic means and generate imaging data;
An imaging apparatus comprising: a microlens array corresponding to at least one imaging means among the plurality of imaging means. The imaging apparatus according to claim 1, wherein the number of microlenses constituting the microlens array is at least two or more. The imaging apparatus according to claim 2, wherein the light having different spectral characteristics is at least R, G, and B. 4. The imaging device according to claim 2, wherein the light having different spectral characteristics is at least Y, M, and C. 4. Viewpoint information interpolation means for interpolating viewpoint information of imaging data generated by the imaging means corresponding to a microlens array having a relatively large number of microlenses, or the imaging means in which no microlens array is disposed;
Parallax image interpolating means for interpolating parallax images of imaging data generated by the imaging means corresponding to a microlens array having a relatively small number of microlenses;
Color imaging data synthesizing means for color-synthesizing imaging data generated by the viewpoint information interpolation means and imaging data generated by the parallax image interpolation means;
The imaging apparatus according to claim 1, further comprising: The imaging device according to any one of claims 1 to 5, wherein the microlens array is disposed on an imaging plane of the imaging lens. In the imaging means,
The imaging means in which the corresponding microlens array is disposed is disposed on the imaging surface of the microlens constituting the microlens array,
The imaging apparatus according to claim 6, wherein an imaging unit in which a corresponding microlens array is not arranged is arranged on an imaging surface of the imaging lens. The imaging device according to claim 1, wherein the spectroscopic unit is a color separation prism. An imaging lens, a spectroscopic unit that splits light that has passed through the imaging lens into at least two light beams having different spectral characteristics, and two or more light beams that are split by the spectroscopic unit are received to generate imaging data. And a microlens array corresponding to at least one imaging unit among the plurality of imaging units.
A viewpoint information interpolation step of interpolating viewpoint information of imaging data generated by the imaging unit corresponding to a microlens array having a relatively large number of microlenses, or the imaging unit in which no microlens array is disposed;
A parallax image interpolation step of interpolating a parallax image of imaging data generated by the imaging unit corresponding to a microlens array having a relatively small number of microlenses;
An imaging apparatus control method comprising: imaging data generated by the viewpoint information interpolation step; and color imaging data synthesis step of performing color synthesis on the imaging data generated by the parallax image interpolation step. An imaging lens, a spectroscopic unit that splits light that has passed through the imaging lens into at least two light beams having different spectral characteristics, and two or more light beams that are split by the spectroscopic unit are received to generate imaging data. A program for causing a computer to control the control of an imaging device including a plurality of imaging units and a microlens array corresponding to at least one imaging unit among the plurality of imaging units,
The computer,
Viewpoint information interpolation means for interpolating viewpoint information of the imaging data generated by the imaging unit corresponding to a microlens array having a relatively large number of microlenses, or the imaging unit in which no microlens array is disposed;
Parallax image interpolating means for interpolating parallax images of imaging data generated by the imaging unit corresponding to a microlens array having a relatively small number of microlenses;
A program that functions as a color imaging data combining unit that performs color synthesis on imaging data generated by the viewpoint information interpolation unit and imaging data generated by the parallax image interpolation unit.

Images