JP2015216428A - Compound-eye imaging apparatus, image processing apparatus, and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a compound-eye imaging apparatus capable of performing an image processing which generates an image with a desired high resolution in a short time.SOLUTION: The apparatus comprises: plural individual-eye imaging optical systems; an imaging device 60 which forms plural object images corresponding to the individual-eye imaging optical systems; an image generating processor 91b which processes plural individual eyes images corresponding to the plural object images outputted from the imaging device 60; a resolution power storage unit 92a where resolution power information relating to the resolution power of the plural individual-eye imaging optical systems is stored in advance; and an individual-eye selector 91a which selects an individual eye image to use for the processing by the image generating processor 91b, on the basis of the stored resolution power information.

Description

  The present invention relates to a compound eye imaging device that performs image processing such as super-resolution processing and distance measurement, an image processing device incorporated in the compound eye imaging device, and an image processing method using the compound eye imaging device.

  In order to meet the demand for thin imaging optical systems, a compound eye imaging apparatus having a plurality of single-eye imaging optical systems, an image processing method corresponding to the compound eye imaging apparatus, and the like have been proposed. Specifically, the same subject is photographed by a plurality of single-eye imaging optical systems, and a plurality of low-resolution images output from the imaging device are combined by image processing to output a single high-resolution image. It is a compound eye imaging device that performs resolution processing. In addition, since single-eye images taken by a plurality of single-eye imaging optical systems have parallax with each other, distance measurement can also be performed. Here, the image quality of the high-resolution image generated by the super-resolution processing greatly depends on the individual image quality of the plurality of low-resolution images to be synthesized. Therefore, when a single-eye image with poor image quality is included in a plurality of low-resolution images, the image quality after super-resolution processing also deteriorates. In distance measurement, corresponding point search between single-eye images having parallax is performed. However, if there is a single-eye image with poor image quality, the accuracy of the corresponding point search is lowered, and the distance accuracy is also lowered. Therefore, as these pre-processing, for example, there is the following technique for selecting a single-eye image with good image quality.

  As pre-processing such as super-resolution processing, there is a method of reducing individual images based on spatial frequency information (see Patent Document 1). Specifically, first, an average spatial frequency distribution (average value) of all the single-eye images is calculated from the spatial frequency distribution of each single-eye image. Thereafter, the difference between the spatial frequency distribution of each individual eye image and the average value is taken, and the individual image whose difference value is equal to or greater than a certain threshold value is deleted. In other words, in the method of Patent Document 1, an image is selected after processing a captured image.

  As another preprocessing such as super-resolution processing, there is a method of using an image with good image quality as a reference image instead of limiting the single-eye image to be used (see Patent Document 2). In this method, high-frequency components are extracted without using Fourier transform. For example, edge extraction using a Laplacian filter is performed, and an integrated value of the output values after extraction is compared to select an image with a small amount of blur and use it as a reference image during super-resolution processing.

  However, in the preprocessing of Patent Document 1, generally, it takes time to calculate the spatial frequency distribution, that is, Fourier transform, and also requires processing for comparing average values and differences, which is an adverse effect of speeding up the processing. . Further, in the preprocessing of Patent Document 2, since the spatial filtering process is performed, the calculation speed can be shortened as compared with the case of performing the Fourier transform, but an image having a large blur amount is finally used for the super-resolution process. There is concern about image quality degradation.

JP 2009-260843 A JP 2009-194896 A

  An object of the present invention is to provide a compound eye imaging apparatus capable of image processing for generating a desired high resolution image in a short time.

  It is another object of the present invention to provide an image treatment device incorporated in the compound eye imaging device and an image processing method using the compound eye imaging device.

  In order to solve the above problems, a compound eye imaging apparatus according to the present invention includes a plurality of single-eye imaging optical systems, an imaging element on which a plurality of object images corresponding to the single-eye imaging optical system are formed, and an output from the imaging element. An image generation processing unit that processes a plurality of single-eye images corresponding to a plurality of object images, a resolving power storage unit that stores resolving power information related to the resolving power of a plurality of single-eye imaging optical systems, and stored resolving power information And a single eye selection unit for selecting a single eye image to be used in the image generation processing unit.

  In the above compound-eye imaging device, the resolution of each single-eye imaging optical system is not determined at the stage of assembling the optical system and the imaging device, but processing the single-eye images after shooting to determine the image quality of each single-eye image. And resolving power information for selecting and determining a single eye image to be used during image processing is stored in advance. By grasping in advance the superiority or inferiority of the resolving power of each single-eye imaging optical system, preprocessing for selecting a single-eye image after shooting is unnecessary, and the entire image processing can be speeded up. In addition, by eliminating single-eye images with low resolving power in advance, single-eye images used for super-resolution processing, etc. are limited. The decrease can be suppressed. Further, in a bright optical system having a smaller F-number, the depth of focus becomes shallow. Therefore, even if the variation in resolving power with respect to the variation in focus position between the single-eye imaging optical systems increases, it is possible to select a single-eye image. Image quality can be maintained.

  In a specific aspect or embodiment of the present invention, the resolution information is one of a modulation transfer function (MTF) and a spatial frequency response (SFR). In this case, the resolving power of the compound-eye imaging optical system can be quantified by measuring MTF or SFR.

  In another aspect of the present invention, the resolution information is associated with the image side distance from the single-eye imaging optical system to the imaging surface of the image sensor, and the single eye to be used is based on the resolution data corresponding to the image side distance. Select an image. In this case, it is possible to obtain a relatively good image when shooting subjects having various shooting distances to the subject at once. Note that the image side distance may be determined in advance or may be changed every time shooting is performed. When the image side distance is determined in advance, the single-eye imaging optical system to be selected is fixed. In addition, when the image-side distance changes at every shooting, the movement amount in the optical axis direction of the single-eye imaging optical system is used as related to the image-side distance from the single-eye imaging optical system to the imaging surface of the image sensor. It is done.

  According to still another aspect of the present invention, an attention subject recognition unit that recognizes an attention subject is provided. Here, the subject of interest means a subject that the user wants to focus on. In this case, by recognizing the subject of interest, the corresponding image height of the subject and the shooting distance to the subject can be grasped, and the resolving power data to be referred to can be more adapted to the subject of interest.

  In still another aspect of the present invention, the resolution information is associated with the corresponding image height of the subject of interest, the corresponding image height is obtained, and the single eye image to be used is determined based on the resolution data corresponding to the corresponding image height. select. In this case, resolving power data to be referred to by grasping the corresponding image height can be made more appropriate, and a good image quality of the subject of interest can be obtained.

  In still another aspect of the present invention, the resolution information is associated with the shooting distance to the subject of interest, acquires the shooting distance, and selects a single-eye image to be used based on the resolution data corresponding to the shooting distance. . Here, the shooting distance means a subject distance. In this case, even when a plurality of subjects at different distances are simultaneously photographed, a good image quality is obtained according to the photographing distance to the subject by using an optimal single-eye image according to the photographing distance to the subject. be able to. Note that the shooting distance to the subject of interest may be determined in advance or may be changed every time shooting is performed.

  In order to solve the above problems, an image processing apparatus according to the present invention uses a plurality of first image data obtained from an image detection unit having a plurality of single-eye imaging optical systems, and has a higher resolution than the first image data. An image processing apparatus that generates second image data, a resolution storage unit that stores in advance resolution information associated with a plurality of single-eye imaging optical systems corresponding to the plurality of first image data, and a plurality of first image data Based on resolution information stored in advance from image data, a single-eye selection unit that selects first image data to be used, and image generation that generates second image data from the first image data selected by the single-eye selection unit A processing unit.

  In the image processing apparatus, by preliminarily grasping the superiority or inferiority of the resolving power of each single-eye imaging optical system, preprocessing for selecting a single-eye image after photographing is unnecessary, and the entire image processing can be speeded up. In addition, by eliminating single-eye images with low resolving power in advance, single-eye images used for super-resolution processing, etc. are limited. The decrease can be suppressed.

  In order to solve the above problems, an image processing method according to the present invention uses a plurality of first image data obtained from an image detection unit having a plurality of single-eye imaging optical systems, and has a higher resolution than the first image data. An image processing method for generating second image data, based on resolving power information associated with a plurality of single-eye imaging optical systems corresponding to first image data stored in advance from a plurality of first image data. Selecting and capturing the first image data to be used, and an image generating step for generating second image data from the plurality of first image data selected in the single image capturing step. Prepare.

  In the above-described image processing method, by preliminarily determining the superiority or inferiority of the resolving power of each single-eye imaging optical system, preprocessing for selecting a single-eye image after shooting is unnecessary, and the overall image processing can be speeded up. In addition, by eliminating single-eye images with low resolving power in advance, single-eye images used for super-resolution processing, etc. are limited. The decrease can be suppressed.

1 is a block diagram illustrating a compound eye imaging device according to a first embodiment of the present invention. (A) And (B) is the top view and sectional drawing explaining an image detection part among the compound eye imaging devices shown in FIG. (A)-(C) are the figures explaining the variation in the characteristic between the single-eye imaging optical systems. It is a block diagram explaining circuit parts, such as an image processing circuit, of the compound eye imaging device shown in FIG. It is a flowchart showing the flow of processing in the image processing circuit. It is a top view explaining the example of arrangement | sequence of the sensor area | region in an optical sensor. It is a figure explaining the Gaussian filter process among image processes. (A)-(C) is a figure which illustrates pixel position shift estimation processing notionally. (A) And (B) is a figure for demonstrating the specific example of a subpixel positional offset estimation process. It is a figure showing the flow of the super-resolution process in step S15 of FIG. It is a figure explaining the deterioration information used by FIG.10 S33. It is a figure showing the specific example of deterioration information. It is a block diagram explaining circuit parts, such as an image processing circuit, among compound eye imaging devices of a 2nd embodiment. It is a flowchart showing the flow of processing in the image processing circuit shown in FIG. It is a block diagram explaining circuit parts, such as an image processing circuit, among compound eye imaging devices of a 3rd embodiment. 16 is a flowchart showing a process flow in the image processing circuit shown in FIG. 15. It is a figure explaining the some to-be-photographed object in an imaging | photography range. It is a figure explaining the data table of the resolution used in the image processing circuit shown in FIG.

[First Embodiment]
[Composite imaging device]
As illustrated in FIG. 1, the compound eye imaging apparatus 300 according to the first embodiment includes an image detection unit 100, a drive processing unit 81, an interface 82, and a display 83. Here, the interface 82 and the display 83 are not essential and may be omitted.

  The image detection unit 100 includes a lens array stack 30 that forms a large number of subject images in parallel, and an image sensor 60 that detects a number of subject images formed by the lens array stack 30 in parallel. In the present embodiment, the compound-eye imaging apparatus 300 is assumed to be a pan-focus (fixed focal length) system, and the distance between the lens array stack 30 and the imaging element 60 is fixed. The lens array laminate 30 is a two-dimensional array of single-eye imaging optical systems 30a having a similar structure in a matrix. The image sensor 60 converts each image formed on the sensor region (photoelectric conversion surface) 61 constituting the image sensor 60 into an electrical signal and outputs the electrical signal to the drive processing unit 81. The drive processing unit 81 processes the input signal based on a predetermined processing program stored in a ROM or the like included in a storage unit provided in the drive processing unit 81, and reconstructs each image into one image. . Thereafter, the drive processing unit 81 outputs one reconstructed image to the display 83 or the like via the interface 82. Further, the drive processing unit 81 temporarily stores various calculation results when executing processing based on the processing program in a RAM or the like included in a built-in storage unit. As the image reconstruction processing by the drive processing unit 81, for example, not only a process of cutting out a necessary rectangular area from each image, but also a process of reconstructing an image from the cut out rectangular image based on each parallax information and the like. Etc. are included. The image reconstruction processing includes image processing using a super-resolution method. Here, the super-resolution processing refers to processing for obtaining one high-resolution image by image processing from images of the same field of view formed by individual lenses.

(Image detector)
2A and 2B are a plan view and a cross-sectional view for conceptually explaining the image detection unit 100 in the compound eye imaging apparatus 300 shown in FIG. As illustrated, the image detection unit 100 includes a lens array stack 30, an image sensor 60 having a sensor region 61 corresponding to each individual eye imaging optical system 30 a constituting the lens array stack 30, and the image sensor 60. Are mounted on a substrate SB.

  The illustrated lens array stacked body 30 is a stacked body in which a plurality (two in the illustrated example) of lens arrays 10 and 20 are stacked, and is used as a compound eye optical system. In the following description, the lens array stack 30 may be referred to as a lens array. These first and second lens arrays 10 and 20 are square plate-like members extending in parallel to the XY plane, and are stacked in the Z-axis direction perpendicular to the XY plane and joined to each other.

  The lens array stack 30 is housed in a rectangular frame-shaped holder 50 in a state of facing the image sensor 60. In the lens array stacked body 30, the first lens array 10 on the object side is an integrally molded product made of a light-transmitting thermoplastic resin that is an optical material, in other words, an integrated product made of a thermoplastic resin. The first lens array 10 has a rectangular outline when viewed from the central axis AX direction or the Z-axis direction. Here, the central axis AX indicates an axis at the center of an array region (lens portion array region LR) of a plurality of lens units 10a described later. The first lens array 10 has a plurality of lens portions 10a each of which is an optical element, a support portion 10b that supports the plurality of lens portions 10a from the periphery, and a rectangular frame shape that extends in a strip shape outside the support portion 10b. And an edge portion 10r. The plurality of lens portions 10a constituting the first lens array 10 are two-dimensionally arranged on square lattice points (16 × 4 × 4 in the illustrated example) arranged in parallel to the XY plane. Each lens unit 10a has a first optical surface 11a that is convex on the first main surface 10p on the object side, and a second optical surface 11b that is convex on the second main surface 10q on the image side. The first and second optical surfaces 11a and 11b are, for example, aspherical surfaces. The support portion 10b is a flat plate-like portion and includes a plurality of peripheral portions 10c so as to surround each lens portion 10a. A square frame-shaped edge portion 10r on the periphery or outside in the lateral direction of the support portion 10b is a portion for joining the first lens array 10 to the second lens array 20.

  The second lens array 20 on the image side is an integrally molded product made of a thermoplastic resin and has a rectangular outline when viewed from the direction of the central axis AX. The second lens array 20 has a plurality of lens portions 20a each of which is an optical element, a support portion 20b that supports the plurality of lens portions 20a from the periphery, and a rectangular frame shape that extends in a band shape outside the support portion 20b. And an edge portion 20r. The plurality of lens portions 20a are two-dimensionally arranged on square lattice points (16 × 4 × 4 in the illustrated example) arranged in parallel to the XY plane. Each lens unit 20a has a first optical surface 21a that is concave on the first main surface 20p on the object side, and a second optical surface 21b that is convex on the second main surface 20q on the image side. The first and second optical surfaces 21a and 21b are, for example, aspherical surfaces. The support portion 20b is a flat plate-like portion and includes a plurality of peripheral portions 20c so as to surround each lens portion 20a. A square frame-shaped edge portion 20r on the periphery or outside in the lateral direction of the support portion 20b is a portion for joining the second lens array 20 to the first lens array 10.

  The lens arrays 10 and 20 are sequentially stacked by a machine or a manual operation so that they are positioned relative to each other by their own structure, their own weight, electrostatic force, or the like. That is, self-alignment is performed. Further, when the lens arrays 10 and 20 are stacked, the lens arrays 10 and 20 are bonded or bonded together by supplying, for example, a photocurable resin between the edge portions 10r and 20r and then curing the photocurable resin. By such bonding or adhesion, a lens array laminate 30 including a large number of single-eye imaging optical systems 30a arranged two-dimensionally in a matrix is obtained. The optical axis OA of each single-eye imaging optical system 30a is parallel to the central axis AX of the entire area of the plurality of lens portions 10a and 20a.

  Between the first lens array 10 and the second lens array 20, a thin light-shielding first diaphragm plate 41 is disposed that is sandwiched and fixed between the pair of support portions 10b and 20b. Although a detailed description is omitted in the first diaphragm plate 41, a large number of openings are formed corresponding to the respective lens portions 10a. On the image side of the second lens array 20, a thin light-blocking second diaphragm plate 42 fixed to the support portion 20b by bonding or the like is disposed. Although a detailed description is omitted, the second aperture plate 42 has a large number of openings corresponding to the lens portions 20a and the like.

  The holder 50 is an integrally molded product made of a thermoplastic resin, and has a ceiling part 50a having a rectangular outline when viewed from the central axis AX direction of the lens array laminate 30, and a ceiling part arranged circumferentially along the outline. A wall-like leg portion 50b extending in the direction of the central axis AX from 50a. The holder 50 is fixed on the substrate SB by adhering the entire bottom surface of the leg portion 50b in a state where the lens array stacked body 30, the imaging device 60, and the like are disposed so as to be surrounded by the ceiling portion 50a and the leg portion 50b. An entrance diaphragm plate 45 fixed by adhesion to the holder 50 or the like is disposed on the object side of the holder 50 that houses the lens array stack 30. Although detailed description is omitted in the ceiling portion 50a and the entrance diaphragm plate 45 of the holder 50, a large number of openings are formed corresponding to the lens portions 10a and the like. In addition, by holding the lens arrays 10 and 20 with the holder 50, a space where the compound eye optical system (lens array stacked body 30) faces the image sensor 60 is secured even if another member is provided around the image sensor 60. be able to.

  The image sensor 60 is mounted on the substrate SB. The sensor region 61 provided in the image sensor 60 is a two-dimensional array of a plurality of pixels that perform photoelectric conversion, and is disposed in a region corresponding to each single-eye imaging optical system 30a on the image sensor 60. Alternatively, it may be arranged on the entire image sensor 60. In addition to the image sensor 60, a signal processing unit and the like are mounted on the substrate SB.

  FIGS. 3A to 3C are diagrams for explaining the variation between the single-eye imaging optical systems 30 a constituting the lens array stacked body 30. In the case of the single-eye imaging optical system 30a shown in FIG. 3A, the modulation transfer function (MTF) or the spatial frequency response (SFR) peaks at a position where the defocus is zero for the lens array stack 30 (that is, the imaging surface). For example, it exceeds the predetermined allowable level LP necessary for obtaining the desired effect in the super-resolution processing. In the case of the single-eye imaging optical system 30a shown in FIG. 3B, the MTF or SFR has a peak in a state where the defocus is greatly deviated from the zero position, and is below the predetermined allowable level LP. ing. In the case of the single-eye imaging optical system 30a shown in FIG. 3C, the MTF or SFR has a peak in a state where the defocus is slightly deviated from the zero position, but the predetermined allowable level LP is not exceeded. It has exceeded. That is, the MTF or SFR characteristic of each single-eye imaging optical system 30a has a similar waveform or pattern, but is unique to each. Actually, not only the peak position of the MTF or SFR shifts in the defocus axis direction or the direction of the MTF value or SFR value, but also the characteristic width or peak width increases or decreases. Furthermore, although the peak position and characteristic width of MTF or SFR change depending on the image height, the degree of change varies depending on the single-eye imaging optical system 30a.

  Ideally, it is desirable that the single-eye imaging optical system 30a constituting the lens array stack 30 has the same MTF characteristic or SFR characteristic. However, due to the manufacturing process of the lens array laminate 30, it is not easy in terms of cost and technology to precisely match the MTF characteristics or SFR characteristics of all the single-eye imaging optical systems 30a. Furthermore, when the lens array stack 30 is positioned and fixed with respect to the image sensor 60, a slight tilt error, a lateral shift error, etc. occur, and the difference in MTF characteristics or SFR characteristics between the individual eye imaging optical systems 30a. Tend to spread. Therefore, in the compound-eye imaging apparatus 300 of the present embodiment, a certain degree of characteristic difference exists between the single-eye imaging optical systems 30a, and image processing or the like that makes it difficult for such a characteristic difference to affect image reconstruction. To do.

Hereinafter, the case where MTF is used as the resolution of the single-eye imaging optical system 30a constituting the lens array stack 30 will be described. MTFn, which is the resolution of the n-th single-lens imaging optical system 30a constituting the lens array stack 30, has a corresponding image height of the main subject as h, a shooting distance (subject distance) to the main subject as D, and a single eye. Assuming that the image side distance from the imaging optical system 30a to the detection surface is fBn, the following expression MTFn = MTF {h, D, fBn} (1)
Given in. Therefore, if information about the resolving power MTFn is acquired in advance for each single-eye imaging optical system 30a, the single-eye imaging optical system 30a in a high-resolution state is selected and operated according to the situation in which the lens array stack 30 is used. The reliability at the time of image reconstruction such as super-resolution can be improved. That is, not all the single-eye imaging optical systems 30a are used, but only the single-eye imaging optical systems 30a having a resolution of a certain level or higher except for the single-eye imaging optical systems 30a whose resolution has deteriorated partially. By performing the configuration process, a high-quality image can be obtained. For example, among the single-eye imaging optical systems 30a shown in FIGS. 3 (A) to 3 (C), the single-eye imaging optical systems 30a shown in FIGS. 3 (A) and 3 (C) in which the MTF exceeds a predetermined allowable level LP. Is selected and image processing is performed. Information relating to the resolving power MTFn of each single-eye imaging optical system 30a can be stored or stored in advance as a data table, for example, and this data table includes the corresponding image height h, the photographing distance D to the main subject, and the individual eye. It is assumed that n resolving power MTFn having the image side distance fBn from the imaging optical system 30a to the detection surface as a parameter is included. In the case of a pan-focus system such as the compound eye imaging apparatus 300 of the first embodiment, the shooting distance D to the main subject is determined by a design value, and the image side distance fBn from the single-eye imaging optical system 30a to the detection surface is also a single eye. It is determined for each imaging optical system 30a. Therefore, before the final product, the three positional relationships of the resolution evaluation chart, the single-eye imaging optical system 30a, and the sensor region 61 are uniquely determined in the adjustment measurement for obtaining the resolution information in advance. The resolution evaluation chart used here may be a chart that can measure the resolution on the high image height side as well as on the axis. By measuring the photographed resolution evaluation chart for each single-eye imaging optical system 30a, for example, the on-axis performance for each single-eye imaging optical system 30a can be evaluated. A method of selecting the single-eye imaging optical system 30a used for image reconstruction will be described in detail later.

[Circuit parts such as drive processing unit]
With reference to FIG. 4, the circuit which comprises the compound eye imaging device 300 of FIG. 1 is demonstrated. In the image detection unit 100, the image sensor 60 includes an optical sensor 63 and an AD conversion unit 65. The optical sensor 63 includes a plurality of sensor regions 61 that perform photoelectric conversion corresponding to the plurality of single-eye imaging optical systems 30a. As already mentioned, the sensor area 61 can be a separate individual image sensor, but can also be a partial area of a single image sensor. The AD converter 65 AD-converts detection signals from the sensor regions 61 at least partially in parallel. The image sensor 60 includes a driving logic circuit and the like, but detailed description thereof is omitted.

  The drive processing unit 81 includes an image processing unit 91, a storage unit 92, an image output unit 93, and a control unit 94. The image processing unit 91 includes a single eye selection unit 91a and an image generation processing unit 91b. The single-eye selection unit 91a is a part that selects a plurality of single-eye imaging optical systems 30a constituting the lens array stacked body 30 that satisfy a predetermined criterion based on information from the control unit 94. The single eye selection unit 91a has a role of selecting image data from the sensor region 61 corresponding to the selected single-eye imaging optical system 30a. The image generation processing unit 91b performs image reconstruction, specifically, super-resolution processing, using the image data from the selected sensor region 61 and the data stored in the storage unit 92. The storage unit 92 includes a resolving power storage unit 92a that stores resolving power information of each single-eye imaging optical system 30a. Here, the resolving power information includes resolution information such as MTFn described with reference to FIGS. The resolving power information is obtained by measuring the resolving power of each single-eye imaging optical system 30a at the stage of assembling the lens array laminate 30 and the image sensor 60, and is stored in the resolving power storage unit 92a. Since the compound-eye imaging apparatus 300 of the present embodiment is configured to assume pan focus, the resolving power storage unit 92a stores the on-axis performance (MTF) of each single-eye imaging optical system 30a with respect to the design subject distance of each single-eye imaging optical system 30a. {H, D} is stored, In the present embodiment, the corresponding image height h and the shooting distance D are fixed for each single-eye imaging optical system 30a, and the image output unit 93 is an image processing unit 91. The reconstructed image obtained by the process is output at a predetermined timing based on an instruction from the control unit 94. The control unit 94 includes a drive processing unit including an image processing unit 91, a storage unit 92, an image output unit 93, and the like. The operation of 81 is comprehensively controlled.

[Image reconstruction processing]
The operation of the compound-eye imaging apparatus 300 will be described with reference to FIG. 5 (see Japanese Patent Application Nos. 2013-249608 and 2013-249607 for details of the steps after the pre-processing).

[Image reconstruction-Capture single image]
The drive processing unit 81 of the compound-eye imaging apparatus 300 monitors imaging conditions, selects a specific single-eye imaging optical system 30a based on the imaging conditions, and causes an image signal to be captured from the corresponding sensor area 61. (Step S11). In other words, among the detection signals captured by the optical sensor 63, image signals from the selected k (k is 4 × 4 or less) sensor regions 61 are AD-converted by the AD conversion unit 65, and the image of the drive processing unit 81 is displayed. The data is input to the processing unit 91. Here, for example, a low-resolution image (first pixel data) of about 500 × 500 pixels is input. The single eye selection unit 91a of the image processing unit 91 selects only single eye images having a threshold value or more based on the resolution information stored in advance in the resolution storage unit 92a. The single eye image is selected every time shooting is performed due to, for example, the use environment. The single-eye image need not be selected every time shooting is performed, and may be selected in advance when the resolving power is measured. In this case, the selected single-eye imaging optical system 30a is stored as setting information in the resolving power storage unit 92a.

  FIG. 6 is a conceptual plan view for explaining an arrangement example of the sensor regions 61 in the optical sensor 63. Although there are 4 × 4 sensor areas 61, only image signals from the selected k sensor areas 61 are used. Specifically, for example, image signals from one sensor region 61 x out of 4 × 4 sensor regions 61 are not used, and image signals from the remaining 15 sensor regions 61 are used.

[Image reconstruction-pre-processing]
Next, the image processing unit 91 of the drive processing unit 81 performs preprocessing on the image signals from the k sensor regions 61 (step S12). The pre-processing considers the case where each pixel constituting the sensor region 61 corresponding to the single-eye photoelectric conversion unit is provided with a color filter for color selection, and the input image to the image processing unit 91 is It is assumed that each pixel has luminance information for only a specific color. In such a case, by performing appropriate smoothing, it is possible to avoid difficulty in correcting a positional deviation described later. Specifically, when a GRBG Bayer-type filter is attached to the pixels constituting the sensor region 61, a 3 × 3 Gaussian filter (see FIG. 7) is applied to the image signal as signal processing. A 1-channel image signal is created for each eye. In such Gaussian filter processing for signal processing, regardless of the pixel position of the GRBG of interest, all the pixels after the filter processing have the same composition ratio of G / 2 + R / 4 + B / 4 as the original pixels. It will be synthesized. This makes it easy to perform template matching, which will be described later, using all the filtered pixels. Such pre-processing filter processing is not limited to the Gaussian filter processing, but i × j (i and j are integers of 3 or more) having weighting coefficients that give the same color composition ratio. Filtering can be used. At this time, since the blur amount of the image changes depending on the filter coefficient, an appropriate one is selected from a plurality of filters in accordance with the characteristics of the input image.

  The pixels constituting the sensor region 61 are not limited to the RGB three-color Bayer array. That is, not only an input image corresponding to an RGB three-color Bayer array, but also an input image corresponding to a four-color pixel array different from 2 × 2 pixels, such as WYRIr or RGBIr, for example, the same smoothing filter processing The same smoothing filter processing is possible even with an input image of 3 colors with 3 × 3 pixels.

  In the above description, an image signal of one channel is obtained by a Gaussian filter or the like. However, RGB three-color images may be individually generated by demosaic processing.

[Image reconstruction-misregistration estimation processing]
Next, the image processing unit 91 performs pixel position deviation estimation processing (step S13) and subpixel deviation estimation processing (step S14) on the image data obtained in step S12. In the pixel position shift estimation process (step S13), the position shift approximate estimation process and the position shift detail estimation process are executed, and the shift amount in pixel units (integer pixel units) is estimated. In the sub-pixel shift estimation process (step S14), the shift amount in sub-pixel units (decimal pixel units) for which the rough estimation is improved more accurately is estimated.

  8A to 8C are diagrams for conceptually explaining the pixel position deviation estimation processing. FIG. 8A shows the reference image IG0 from the sensor region 61 closer to the center of the optical sensor 63 shown in FIG. 6, and FIGS. 8B and 8C show the reference image IG0 of the optical sensor 63 closer to the center. The reference image IG1 from any one of the sensor regions 61 other than the sensor region 61 is shown.

  In the previous positional deviation approximate estimation process in the pixel positional deviation estimation process (step S13), first, image information is cut out from a specific area (window) A in the reference image IG0 shown in FIG. Then, the pixel position is searched with coarse accuracy while moving the search region (window) B in the direction of arrow β in the reference image IG1 shown in FIG. For example, a position shifted by the amount of misregistration (m × p, m × p) with respect to the specific area A (m means every m pixels, for example, 4 and p corresponds to p searches. For example, the search region B is moved stepwise to perform template matching, and a position shift with the highest degree of matching is searched. At this time, the direction of the arrow β for moving the search area B corresponds to the direction in which the parallax of the sensor area 61 shown in FIG. 8B appears. That is, the direction of the arrow β for moving the search area B is appropriately set according to the place where the sensor area 61 is arranged. Here, for example, NCC (Normalized Cross Correlation) is used for template matching. However, the template matching is not limited to NCC, but may be another method such as SAD (Sum of Absolute Difference) or SSD (Sum of Squared Difference).

  Subsequently, in the positional displacement detail estimation processing later in the pixel positional displacement estimation processing (step S13), positional displacement estimation is performed by improving the rough estimation in step S13 more accurately. As shown in FIG. 8 (C), as a result of the above-described misregistration rough estimation process, a process for searching for corresponding points in a plane with accuracy of one pixel around the center pixel of the window Bta having the highest degree of coincidence is executed. The range of the corresponding point search at this time is, for example, a search for every m pixels (in the above example, every m = 4 pixels in the XY direction) by the approximate position shift estimation process, and this is determined according to this every m pixels. The range to be covered (for example, a range of ± 4 pixels). More specifically, as a result of the above-described approximate positional deviation estimation process, 81 windows are set around each of ± 4 pixels around the center pixel of the window Bta having the highest degree of coincidence, and a template is set for each of these 81 windows. Matching is executed, and the pixel at the center position in the window Btb having the highest degree of matching is obtained as the corresponding point.

  Returning to FIG. 5, next, the image processing unit 91 performs a subpixel shift estimation process (step S <b> 14). In this subpixel shift estimation process (step S14), the position shift estimation is performed in units of subpixels in which the position shift estimation process in step S13 is more accurately improved.

  FIGS. 9A and 9B are diagrams for explaining the subpixel shift estimation processing in step S14. For example, with respect to the reference image IG1 in FIG. 8C, a positional shift in subpixel units is estimated. The degree of coincidence for each pixel in template matching in the specified range C4 of the range of, for example, 3 × 3 pixels centered on the pixel TP corresponding to the standard pixel in the reference image IG1 specified by the positional deviation estimation processing step in step S13. Quadratic surface fitting is performed based on (for example, NCC value) (see FIG. 9B). In the sub-pixel shift estimation process in step S14, as an example, the paper “Robust Super-Resolution Process Based on Pixel Selection Considering Position Shift Amount” (The Institute of Electronics, Information and Communication Engineers, Vol. J92-D, No. 5, pp. .650-660, 2009, May 2009) ”can be employed. That is, as shown in FIG. 9B, the coordinates of the pixel having the highest matching degree among the matching degrees for each pixel are specified as the shift amount. Since pattern matching is performed after setting the search area in this way, processing can be performed at a higher speed than when pattern matching is performed on all images. Note that, instead of performing quadratic surface fitting as in the above paper, subpixel shift estimation processing may be performed by another method such as fitting to a quadratic curve on each of the XY axes.

[Image reconstruction-super-resolution processing (image generation)]
With reference to FIG. 10, the flow of the super-resolution processing in step S15 of FIG. 5 will be described. In FIG. 10, as a specific example, a case where the processing described in the paper “Fast and Robust Multiframe Super Resolution” (IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 13, NO. 10, OCTOBER 2004 page.1327-1344) is performed is shown. The flow of resolution processing is shown.

  As shown in FIG. 10, in step S31, one of the input images is subjected to an interpolation process such as a bilinear method to convert the resolution of the input image to a high resolution that is the resolution after the super-resolution process. Thus, an output candidate image as an initial image is generated.

  In step S32, a BTV (Bilateral Total Variation) amount for robust convergence even if there is noise is calculated.

  In step S33, the generated output candidate images are compared with k input images (16 or less, for example, 15 images), and a residual is calculated. That is, in step S33, the generated output candidate image is converted into an input image size based on each input image and its deterioration information (information indicating the relationship between the image after super-resolution and the input image). After the resolution is lowered (step S41 in FIG. 11), the difference from the input images for k (for example, 15) is calculated and recorded (step S42 in FIG. 11). Then, the difference is returned to the size after the super-resolution processing (step S43 in FIG. 11) and is set as a residual.

  If it is determined in step S34 that an output candidate image needs to be generated, and an output candidate image needs to be generated (Y in step S34), the process returns to step S31 and starts from the output candidate image generated in the previous step S31. The calculated residual and the BTV amount are reduced to generate the next output candidate image.

  The processes in steps S31 to S34 are repeated until the output candidate image converges. When the output candidate image converges (N in step S34), the converged output candidate image is output as the output image after the super-resolution processing. (Step S35).

  The repetition may be a predetermined number of times such as the number of times of convergence (for example, 200 times) approximately enough, or a convergence determination may be made for each series of processing, and may be repeated according to the result. .

  FIG. 11 is a diagram for explaining the deterioration information used in step S33. Deterioration information refers to information representing the relationship of each input image with respect to a high-resolution image after super-resolution processing, and is represented, for example, in a matrix format. The deterioration information includes a shift amount, a downsampling amount, a blur amount, and the like of each of the input images estimated in step S14.

  As shown in FIG. 11, the deterioration information is defined by a matrix indicating the conversion when each of the input image and the high-resolution image after the super-resolution processing is represented by a one-dimensional vector expression. The compound-eye imaging apparatus 300 calculates parameters used for the super-resolution processing based on, for example, the positional deviation estimated in the estimation processing in steps S13 and S14, and incorporates it as deterioration information.

  FIG. 12 is a diagram illustrating a specific example of deterioration information. The compound-eye imaging device 300 is estimated to have a pixel shift amount of 0.25 pixel, and the deterioration information as shown in FIG. 12 when the down-sampling amount is 1/4 in each of the vertical and horizontal directions. Is specified.

  Note that the super-resolution processing in step S15 is not limited to the processing shown in FIG. 10, but is reconfigurable super-resolution processing that generates one image from a plurality of input images. Other processes may be employed if any. The constraint term calculated in step S32 may be replaced with another value such as a 4-neighbor Laplacian instead of the BTV amount.

  Returning to FIG. 5, after the above super-resolution processing (step S <b> 15) is completed, the high-resolution image (second image data) generated by the image processing unit 91 is externally provided via the image output unit 93, i.e., FIG. 1. Are output to the interface 82 (step S16).

  According to the compound-eye imaging apparatus described above, each individual image is not subjected to processing to determine the image quality by processing a single-eye image after shooting, but at the stage of assembling the lens array laminate 30 and the image sensor 60 that are optical systems. The resolving power information for measuring the resolving power of the eye imaging optical system 30a and selecting and determining the single-eye imaging optical system 30a used at the time of image processing is stored in advance. By grasping in advance the superiority or inferiority of the resolving power of each single-eye imaging optical system 30a, preprocessing for selecting a single-eye image after photographing is unnecessary, and the entire image processing can be speeded up. In addition, by eliminating single-eye images with low resolving power in advance, single-eye images used for super-resolution processing, etc. are limited. The decrease can be suppressed. In addition, in a bright optical system having a smaller F-number, the depth of focus becomes shallow. Therefore, even if the variation in resolving power with respect to the variation in focus position between the single-eye imaging optical systems 30a increases, by selecting a single-eye image, Good image quality can be maintained.

[Second Embodiment]
Hereinafter, a compound eye imaging device and the like according to the second embodiment will be described. Note that the compound eye imaging device and the like of the second embodiment is a modification of the compound eye imaging device and the like of the first embodiment, and items that are not particularly described are the same as those of the first embodiment.

[Circuit parts such as drive processing unit]
As shown in FIG. 13, in the second embodiment, the compound eye imaging apparatus 300 has a configuration capable of autofocusing, and includes the AF driving apparatus 70 in the compound eye imaging apparatus 300 shown in FIG. 1. That is, in the compound-eye imaging apparatus 300 of the present embodiment, for example, autofocus is assumed in which the user or the like specifies the subject of interest. Here, the subject of interest means a subject that the user or the like wants to focus on. The subject of interest may be determined manually or automatically.

  The AF drive device 70 includes a target subject recognition unit 71, a focus position determination unit 72, and a focus drive unit 73. The subject-of-interest recognition unit 71 is a portion that recognizes and identifies the subject of interest designated by the user or the like. For example, the user moves the target range to the attention area, and enables to capture information for focusing (focus reference information) in the attention area. The focus position determination unit 72 is a part that fetches focus reference information for the target subject specified by the target subject recognition unit 71 and determines the focus position from this information. The focus position is determined using, for example, a phase difference detection method using a dedicated sensor. In the phase difference detection method, image data from one or more single-eye imaging optical systems 30a themselves may be used. The focus driving unit 73 adjusts the height of the holder 50, that is, the distance between the lens array stack 30 and the image sensor 60 (collective image side distance) according to the focus position determined by the focus position determination unit 72 ( (See FIG. 2B). As the focus driving unit 73, for example, a piezo actuator, a stepping motor, a voice coil motor, or the like is used. The position (focus position) of the single-eye imaging optical system 30a, that is, the image side distance fBn from the single-eye imaging optical system 30a to the detection surface can be arbitrarily changed. Note that the focus position may be changed between several predetermined positions.

  Since the compound-eye imaging apparatus 300 of the present embodiment is configured to assume autofocus, the resolving power storage unit 92a includes the corresponding image height h of the main subject for each individual-eye imaging optical system 30a, and the individual-eye imaging optical system 30a. The image-side distance fBn from the detection surface to the detection surface is used as a variable, and the resolution information associated therewith is stored in advance. The shooting distance D to the main subject is the distance of the subject with the highest on-axis performance in the image side distance fBn from a single-eye imaging optical system 30a to the detection surface.

[Image reconstruction processing]
With reference to FIG. 14, the operation of the compound eye imaging apparatus 300 of the present embodiment will be described.
In the present embodiment, focus setting is performed by the AF driving device 70 (step S17) before the selected single-eye image described in the first embodiment is captured (step S11). First, the target subject recognition unit 71 of the AF driving device 70 specifies the target subject to be focused by the user or the like. Next, the focus position determination unit 72 determines a focus position that matches the determined subject of interest. Based on the determined focus position, the focus driving unit 73 adjusts the image side distance and changes the image side distance fBn from each compound-eye imaging optical system 30a to the detection surface.

  Here, a data table prepared in advance and stored in the resolving power storage unit 92a will be described. In the case of an autofocus system such as the compound eye imaging apparatus 300 of the present embodiment, the shooting distance D to the main subject, the corresponding image height h of the main subject, and the image side distance fBn from the single-eye imaging optical system 30a to the detection surface. Resolution information with these three parameters is required. Therefore, the three positional relationships between the resolution evaluation chart, the single-eye imaging optical system 30a, and the sensor region 61 need to be changed within an arbitrary range. For example, first, the distance between the resolution evaluation chart and the single-eye imaging optical system 30a is fixed, and the image side distance fBn from the single-eye imaging optical system 30a to the detection surface is changed to change the MTF {h, D = D1, fB}. Next, the same measurement is performed by changing the distance between the resolution evaluation chart and the single-eye imaging optical system 30a, and MTF {h, D = D2, fB} is acquired. At this time, the image-side distance fBn from the single-eye imaging optical system 30a to the detection surface may be within a range operated by autofocus. In addition, the distance between the resolution evaluation chart and the single-eye imaging optical system 30a may be changed from the infinity to the closest distance at which photographing can be performed by changing the reciprocal of the distance at equal intervals.

  Returning to FIG. 14, the image signal from the sensor region 61 is AD converted in the AD conversion unit 65 in a state where the focus of the subject of interest is the best while repeating feedback between the AF driving device 70 and the image detection unit 100. And input to the image processing unit 91 of the drive processing unit 81. Along with the captured image, the corresponding image height of the subject of interest and the focus position of the optical system (corresponding to the image-side distance fBn from the single-eye imaging optical system 30a to the detection surface) are input to the image processing unit 91 from the AF driving device 70. The The single-eye selection unit 91a reads resolution information of the single-eye imaging optical system 30a with respect to the image height and the focus position input with reference to the data table stored in the resolution storage unit 92a, and only single-eye images with a certain threshold or more are read. Select. In the subsequent image generation processing unit 91b, only the selected single-eye image is used (step S11).

  Thereafter, similarly to the first embodiment, steps S12 to S16 are performed.

  In the present embodiment, the focus position determination unit is not limited to the phase difference detection method, and a contrast detection method in which the focus position is determined by detecting the contrast of the image output from the compound-eye imaging apparatus 300 may be used. .

[Third Embodiment]
Hereinafter, a compound eye imaging device and the like according to the third embodiment will be described. Note that the compound eye imaging device and the like of the third embodiment are modifications of the compound eye imaging device and the like of the first embodiment, and items that are not particularly described are the same as those of the first embodiment.

[Circuit parts such as drive processing unit]
As shown in FIG. 15, in the third embodiment, the compound eye imaging apparatus 300 has a configuration capable of auto-focusing, and the image shown in FIG. 15 in the drive processing unit 81 of the compound eye imaging apparatus 300 shown in FIG. 1. The processing unit 91 includes a distance measurement unit 91d. 15 mainly describes the system configuration of the autofocus system, but can be applied to a panfocus system as will be described later.

  The compound eye imaging apparatus 300 of the present embodiment is configured to select a single eye image after distance measurement. The distance measuring unit 91d is a part that performs distance measurement on the captured image input to the image processing unit 91. The distance measuring unit 91d measures distances for a plurality of subjects in the captured image. By measuring the distance, main subjects at substantially the same distance can be recognized as a group. In the first and second embodiments described above, the optimum single-eye image is selected with respect to the shooting distance to one main subject. In the third embodiment, the shooting distance to each main subject that was shot is selected. Accordingly, the single-eye image used in the image processing unit 91 can be switched.

  In the resolving power storage unit 92a of the present embodiment, the corresponding image height h of the main subject, the shooting distance D to the main subject, and the image side distance from the single-eye imaging optical system 30a to the detection surface for each single-eye imaging optical system 30a. The resolution information associated with fBn is stored in advance.

[Image reconstruction processing]
With reference to FIG. 16, the operation of the compound eye imaging apparatus 300 of the present embodiment will be described. An image captured using the compound-eye imaging apparatus 300 according to the present embodiment includes a plurality of subjects OB1, OB2, and OB3 as illustrated in FIG. The compound-eye imaging apparatus 300 can obtain, for example, an optimal image for each target subject OB1, OB2, OB3, and obtain a high-resolution image.

  In the present embodiment, the initial focus setting is performed by the AF driving device 70 (step S27) before the selected single-eye image described in the first embodiment is captured (step S11). The focus position is shifted from the initial position, and scanning is performed (steps S28 and S29). As a result, the image signals from all the sensor areas 61 are AD-converted by the AD conversion unit 65 in a state where the focus of the subject of interest OB1, OB2, OB3 in the shooting range is in the best focus, and the image processing unit of the drive processing unit 81 91.

Next, the shooting distance D to each target subject OB1 to OB3 is measured (step S51). Pixel shifts (parallax) in each target subject OB1 to OB3 are calculated from each captured image in which each target subject OB1 to OB3 is in focus, and converted to a shooting distance D to each target subject OB1 to OB3. For example, pixel shift, that is, parallax, is calculated using a single-eye image corresponding to two separate single-eye imaging optical systems 30a located at the diagonal or opposite side midpoint. The shooting distance D to the subject of interest can be obtained by the following equation when the distance between the single-eye imaging optical systems 30a is B, the focal length is f, and the parallax is Z.
D = B × f / Z (2)

  As shown in FIG. 18, the image-side distance fBn from the single-eye imaging optical system 30a to the detection surface is determined by the shooting conditions, and the corresponding image height h and the shooting distance to the target subject for each target subject OB1 to OB3. D is determined. For example, in the case of FIG. 17, the target heights OB1 and OB3 correspond to the image height AR2 of the image heights AR1 to AR3, and the target height OB2 corresponds to the image height AR1.

  Next, using the photographing distance D calculated in step S51, the photographed image is divided into regions corresponding to the plurality of subjects of interest OB1 to OB3 (step S61). Each of the divided areas has a unique resolution characteristic, and it is desirable to select a single eye image suitable for this.

  Thereafter, the single-eye selection unit 91a selects k appropriate single-eye imaging optical systems 30a for each region of the subject of interest OB1 to OB3 focused on the acquired captured image (step S11). That is, the single eye selection unit 91a preliminarily determines the resolving power storage unit 92a from the corresponding image height h of each target subject, the shooting distance D to each target subject, and the image side distance fBn from each single-eye imaging optical system 30a to the detection surface. A single eye image to be used is selected for each region or each pixel of the subject of interest OB1 to OB3 based on the resolution information stored in. The subsequent image generation processing unit 91b performs image processing by switching the single-eye image referred to for each of the subjects of interest OB1 to OB3 or for each pixel.

  Thereafter, similarly to the first embodiment, steps S12 to S16 are performed.

  The distance measurement (step S51) may be performed without using parallax. For example, a focus position that is in focus is searched for in the subject of interest in the captured image. By acquiring a captured image at a position where blurring of the image is reduced, the image side distance fBn from each single-eye imaging optical system 30a to the detection surface and the corresponding image height h of each subject are determined.

  Also, if the subject of interest in the same area is in the same area and exceeds multiple image height thresholds, it should be based on one of the image heights or the average of the image heights. Thus, it may have continuity.

  In the third embodiment, the configuration of the autofocus system has been mainly described. However, the configuration of the panfocus system also operates in the configuration as in the first embodiment.

  In the present embodiment, one image may be obtained in the steps S27 to S29. That is, an image focused on a representative target subject is acquired, and the steps after distance measurement (step S51) are performed. In this case, the subject other than the representative subject of interest can be a slightly blurred image.

  Although the compound eye imaging device according to the embodiment has been described above, the compound eye imaging device according to the present invention is not limited to the above-described example. For example, the arrangement of the lens portions 10a and 20a constituting the lens arrays 10 and 20 and the shape of the optical surfaces 11a and 11b are appropriately changed according to the use or specification of the lens array stack 30 or the compound-eye imaging device 300. can do. For example, the lens units 10a and 20a are not limited to being arranged at 4 × 4 lattice points, but can be arranged at lattice points of 3 × 3, 5 × 5, and the like. The lattice is not limited to a square lattice, and may be a rectangular lattice.

  In the above embodiment, two array lenses are stacked. However, a single-layer array lens may be used instead of stacking. Further, three or more array lenses may be laminated.

  In the above embodiment, the lens arrays 10 and 20 are made of resin, but may be made of glass. The resin is not limited to a thermoplastic resin, and a thermosetting resin, a photocurable resin, or the like can be used. The lens arrays 10 and 20 may be, for example, wafer level lenses in which resin lens portions are two-dimensionally arranged on a glass substrate.

  In the above-described embodiment, the single-eye imaging optical system 30a selected in the single-eye selection unit 91a exceeds the fixed threshold (allowable level LP), but may be a relative threshold. For example, the single-eye imaging optical system 30a may be selected in the order of good resolution. Further, the allowable level LP can be appropriately adjusted according to the purpose.

  In the first and second embodiments, distance measurement may be performed instead of the super-resolution processing (step S15). In this case, if high ranging accuracy is not required, the subpixel shift estimation (step S14) may be omitted.

  DESCRIPTION OF SYMBOLS 30 ... Lens array laminated body, 30a ... Single-eye imaging optical system 10, 20 ... Lens array, 10a, 20a ... Lens part, 10b, 20b ... Support part, 11a, 11b, 21a, 21b ... Optical surface, 50 ... Holder , 60: Image sensor, 61: Sensor area, 63: Optical sensor, 65 ... Conversion unit, 70 ... AF drive device, 71 ... Target object recognition unit, 72 ... Focus position determination unit, 73 ... Focus drive unit, 81 ... Drive Processing unit 82 ... Interface 83 ... Display 91 ... Image processing unit 91a ... Single eye selection unit 91b ... Image generation processing unit 91d ... Distance measuring unit 92 ... Storage unit 92a ... Resolving power storage unit 93 ... Image output unit, 94 ... Control unit, 100 ... Image detection unit, 300 ... Compound eye imaging device, AX ... Central axis, OA ... Optical axis

Claims (8)

A plurality of single-eye imaging optical systems;
An imaging device on which a plurality of object images corresponding to the single-eye imaging optical system are formed;
An image generation processing unit that processes a plurality of single-eye images corresponding to a plurality of object images output from the imaging device;
A resolving power storage section in which resolving power information related to resolving power of the plurality of single-eye imaging optical systems is stored;
A single-eye selection unit that selects a single-eye image to be used for processing in the image generation processing unit based on the stored resolution information;
A compound eye imaging apparatus comprising:   The compound eye imaging apparatus according to claim 1, wherein the resolution information is one of a modulation transfer function and a spatial frequency response. The resolving power information is associated with an image side distance from the single-eye imaging optical system to the imaging surface of the imaging device,
3. The compound eye imaging apparatus according to claim 1, wherein a single eye image to be used is selected based on resolution data corresponding to the image side distance.   The compound eye imaging apparatus according to any one of claims 1 to 3, further comprising an attention subject recognition unit that recognizes an attention subject. The resolving power information is associated with the corresponding image height of the subject of interest,
5. The compound eye imaging apparatus according to claim 4, wherein the corresponding image height is acquired, and a single-eye image to be used is selected based on resolution data corresponding to the corresponding image height. The resolution information is associated with the shooting distance to the subject of interest,
6. The compound-eye imaging apparatus according to claim 4, wherein the imaging distance is acquired, and a single-eye image to be used is selected based on resolution data corresponding to the imaging distance. An image processing apparatus that generates second image data having a higher resolution than the first image data using a plurality of first image data obtained from an image detection unit having a plurality of single-eye imaging optical systems,
A resolving power storage unit preliminarily storing resolving power information associated with the plurality of single-eye imaging optical systems corresponding to the plurality of first image data;
A single-eye selection unit that selects the first image data to be used based on the resolution information stored in advance from the plurality of first image data;
An image generation processing unit for generating the second image data from the first image data selected by the single eye selection unit;
An image processing apparatus comprising: An image processing method for generating second image data having a higher resolution than the first image data by using a plurality of first image data obtained from an image detection unit having a plurality of single-eye imaging optical systems,
The first image data to be used is selected from the plurality of first image data based on resolution information associated with the plurality of single-eye imaging optical systems corresponding to the first image data stored in advance. A single eye image capturing process to be captured;
An image generating step of generating the second image data from a plurality of first image data selected by the single-eye image capturing step;
An image processing method comprising:

Images