JP2012205014A - Manufacturing device and manufacturing method of imaging apparatus, and imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To manufacture an imaging apparatus called as a plenoptic camera so as to appropriately generate a reconstructed image data independent of an individual difference.SOLUTION: An image acquisition part 41 acquires data of a light field image obtained as a result of imaging a light source radiating parallel light or equal diffusion light by an imaging apparatus 2. An error calculation part 43 calculates respective errors of a position and parallelism of a microlens array and an error of respective pitches of a plurality of microlenses generated when the imaging apparatus 2 is generated by using the data of the light field image acquired by the image acquisition part 41 and an ideal parameter acquired by an ideal parameter acquisition part 42. An error information output part 44 outputs information indicating the errors calculated by the error calculation part 43 or the like to the imaging apparatus 2 as error information.

Description

  The present invention relates to an imaging apparatus manufacturing apparatus and manufacturing method called an plenoptic camera, and an imaging apparatus, and more particularly to a technique for manufacturing an imaging apparatus capable of appropriately generating reconstructed image data regardless of individual differences.

In recent years, an imaging device that captures information about the direction distribution of incident light rays, that is, an imaging device called a “plenoptic camera” has been researched and developed (see Patent Document 1).
In the optical system of a plenoptic camera, an extremely small lens (hereinafter referred to as a “micro lens”) is continuously arranged vertically and horizontally between a conventional imaging lens (hereinafter referred to as a “main lens”) and an image sensor. A compound eye lens (hereinafter referred to as a “microlens array”) that is repeatedly arranged is inserted.

The individual microlenses constituting the microlens array distribute the light collected by the main lens to a plurality of pixels in the image sensor according to the angle reached.
That is, if an image focused on the image sensor by each of the individual microlenses is hereinafter referred to as a “sub-image”, image data composed of a collection of a plurality of sub-images is captured as image data. Is output from.
Such a captured image of the plenoptic camera, that is, an image made up of a collection of a plurality of sub-images is hereinafter referred to as a “light field image”.
Thus, the light field image is generated not only by the conventional main lens but also by light incident through the microlens array. For this reason, the light field image has not only two-dimensional spatial information that was also included in the conventional captured image, but also from the image sensor as information that was not included in the conventional captured image. Thus, it has two-dimensional direction information indicating from which direction the light ray has arrived.

Therefore, the plenoptic camera uses such two-dimensional direction information, and after the light field image is captured, the light field image data is used to separate the light field image forward by an arbitrary distance. The image of the surface can be reconstructed.
In other words, even if the plenoptic camera captures a light field image without focusing at a predetermined distance, the data of the light field image is used after the imaging to match the light field image. Data of an image that is captured in focus (hereinafter referred to as “reconstructed image”) can be freely created.

Specifically, a plenoptic camera sets one point on a surface at an arbitrary distance as a point of interest, and light from the point of interest passes through a main lens and a microlens array in any of the image sensors. Calculate whether it is distributed to the pixels.
Here, for example, if each pixel of the imaging device corresponds to each pixel constituting the light field image, the plenoptic camera detects the pixel of interest from among the pixels constituting the light field image. Integrate the pixel values of one or more pixels to which the light is distributed. This integral value becomes the pixel value of the pixel corresponding to the target point in the reconstructed image. In this way, the pixel corresponding to the attention point in the reconstructed image is reconstructed.
The plenoptic camera sequentially sets each pixel (each pixel corresponding to each point on the surface at an arbitrary distance) constituting the reconstructed image as a point of interest, and repeats the above-described series of processing. Thus, the reconstructed image data (a collection of pixel values of each pixel of the reconstructed image) is reconstructed.

Special table 2009-532993

However, considering the case where the plenoptic camera described in Patent Document 1 is mass-produced, it is assumed that manufacturing errors and assembly errors occur when the main lens, the microlens array, and the image sensor are incorporated. .
In this case, considering the case where each of the plurality of plenoptic cameras after imaging captures the subject, the position of each pixel to which the light from the point of interest is distributed at the time of actual imaging is determined by manufacturing error or assembly error. Due to this, deviation occurs for each individual. On the other hand, as the position of each pixel whose pixel value is integrated in order to reconstruct the pixel of interest, the ideal position of each pixel when no manufacturing error or assembly error occurs is the individual difference. Regardless of whether it is adopted.
That is, in each of the plurality of plenoptic cameras, the position of each pixel to which the light from the attention point is distributed during actual imaging and the object whose pixel values are integrated to reconstruct the pixel of the attention point There is a difference between each pixel position for each individual.
When such a shift occurs, it becomes difficult to generate correct reconstructed image data.

  The present invention has been made in view of such a situation, and manufactures an imaging device called a plenoptic camera so that reconstructed image data can be appropriately generated regardless of individual differences. With the goal.

In order to achieve the above object, an apparatus for manufacturing an imaging device of one embodiment of the present invention includes:
An apparatus for manufacturing an imaging apparatus including a main lens, a microlens array including a plurality of microlenses, and an imaging element in an optical system,
A light source for irradiating the imaging device with parallel light or uniform diffused light;
Light field image data consisting of a collection of a plurality of sub-images generated by each of the plurality of microlenses, obtained as a result of the light source emitting parallel light or uniform diffused light being imaged by the imaging device An image acquisition unit for acquiring
Data of an ideal light field image obtained in advance as a result of the light source irradiating parallel light or uniform diffused light being previously imaged by an ideal imaging device, or the ideal imaging device An ideal parameter acquisition unit for acquiring, as an ideal parameter, a parameter obtained from the design value of the optical system;
Using the data of the light field image acquired by the image acquisition unit and the ideal parameter acquired by the ideal parameter acquisition unit, the position and parallelism of the microlens array generated at the time of manufacturing the imaging device An error calculation unit for calculating each error;
An error information output unit that outputs information indicating each error calculated by the error calculation unit or information generated based on each error to the imaging device as error information;
It is characterized by providing.

  According to the present invention, an imaging device called a plenoptic camera can be manufactured so that reconstructed image data can be appropriately generated regardless of individual differences.

It is a schematic diagram which shows the structural example of the manufacturing apparatus of the imaging device which concerns on one Embodiment of this invention. It is a figure which shows the positional relationship of a light source and the optical system of an imaging device in case the manufacturing apparatus of the imaging device of FIG. 1 performs a calibration process. It is a functional block diagram which is a functional structural example of a control apparatus among the manufacturing apparatuses of the imaging device of FIG. 1, and shows the functional structural example which implement | achieves the function which performs a calibration process. 3 is a flowchart illustrating an example of a flow of calibration processing executed by the imaging device manufacturing apparatus in FIG. 1. It is a figure explaining an example of the calculation method of the error of the xy direction of a micro lens array calculated | required by a calibration process. It is a figure explaining an example of the calculation method of the error of the z direction of a micro lens array calculated | required by a calibration process. It is a figure explaining an example of the calculation method of the pitch error of each micro lens calculated | required by a calibration process. It is a figure explaining an example of the calculation method of the parallelism error of a micro lens array and an image sensor which are calculated | required by a calibration process. It is a block diagram which shows the structural example of the hardware of the control apparatus of FIG. It is a block diagram which shows the structure of the hardware of the imaging device which concerns on this invention manufactured with the manufacturing apparatus of the imaging device of FIG. It is a schematic diagram which shows the structural example of an optical system among the imaging devices which have a structure of FIG. 11 is a flowchart for explaining an example of a flow of reconstruction processing executed by the imaging apparatus in FIG. 10.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a schematic diagram illustrating a configuration example of a camera manufacturing apparatus according to an embodiment of an imaging apparatus manufacturing apparatus of the present invention.

The camera manufacturing apparatus 1 is installed in a line for recording initial data and the like in the built-in memory of the image pickup apparatus 2 after assembly, among the production lines of the image pickup apparatus 2. The camera manufacturing apparatus 1 includes a light source 11 and a control device 12 in order to execute a calibration process described later.
The imaging device 2 is manufactured as a plenoptic camera by such a camera manufacturing device 1.

  FIG. 2 is a diagram illustrating a positional relationship between the light source 11 and the optical system of the imaging device 2 when the camera manufacturing apparatus 1 executes a calibration process.

  The light source 11 includes a parallel light irradiation device that irradiates the imaging device 2 with parallel light, and a uniform diffused light irradiation device that irradiates the imaging device 2 with uniform diffusion light.

In the optical system of the imaging device 2, a main lens 21, a microlens array 22, and an imaging element 23 are arranged in that order when viewed from the light source 11 (from the subject).
In the microlens array 22, each of the N microlenses 22-1 to 22-N is regularly and repeatedly arranged.

  The main lens 21 condenses the light emitted from the light source 11 and forms an image on the microlens array 22.

The microlens 22-i (i is an integer value in the range of 1 to N) in the microlens array 22 condenses the light beam incident from the light source 11 through the main lens 21 for each incident direction. Then, a sub-image is formed on the image sensor 23.
That is, in the imaging device 23, a plurality of sub-images are formed by each of the plurality of microlenses 22-1 to 22-N, and a light field image that is an aggregate of the plurality of sub-images is generated.

The image sensor 23 is configured by, for example, a CMOS (Complementary Metal Oxide Semiconductor) type photoelectric conversion element. A subject image (in the example of FIG. 2, an image of the light source 11) is incident on the imaging element 23 via the microlens array 22 from the main lens 21. Therefore, the image sensor 23 photoelectrically converts (captures) the subject image, accumulates the image signal for a predetermined time, and sequentially supplies the accumulated image signal to an AFE (Analog Front End) (not shown) as an analog signal.
The AFE performs various signal processing such as A / D (Analog / Digital) conversion processing on the analog image signal. A digital signal is generated by various signal processing, and is provided as appropriate to a CPU (Central Processing Unit) (not shown) from the AFE as light field image data.

Here, consider a case where the imaging device 2 generates reconstructed image data from light field image data obtained when a general subject (other than the light source 11 in FIG. 2) is imaged.
In this case, as described above, when the imaging device 2 sets one point on a surface at an arbitrary distance as a point of interest, light from the point of interest passes through the main lens 21 and the microlens array 22 and the image sensor 23. To which of the pixels is distributed.
Then, the imaging apparatus 2 estimates and calculates the pixel value of the pixel corresponding to the target point in the reconstructed image by integrating the pixel value in the data of the light field image corresponding to the distributed pixel.
The imaging device 2 generates data of the reconstructed image by executing such estimation calculation for each pixel of the reconstructed image.

  However, in such an estimation calculation of each pixel of the reconstructed image (hereinafter referred to as “reconstruction calculation”), light from the point of interest passes through the main lens 21 and the microlens array 22 and the image sensor 23. If it is not accurately determined which of the pixels to be distributed, noise will appear in the reconstructed image. That is, it becomes difficult to generate correct reconstructed image data.

Here, the following parameters (A) to (J) are used to determine to which pixel in the image sensor 23 the light from the point of interest is distributed via the main lens 21 and the microlens array 22. Is required.
(A) Focal length of main lens 21 (B) Effective diameter of main lens 21 (C) Focal length of each micro lens 22-i (D) Effective diameter of each micro lens 22-i (E) Pixel of image sensor 23 Size (F) Pitch of each micro lens 22-i (G) Positional relationship between main lens 21 and microlens array 22 (H) Positional relationship between microlens array 22 and imaging element 23 (I) Main lens 21 and microlens array (J) Parallelism between the microlens array 22 and the image sensor 23

That is, unless these parameters (A) to (J) are accurately obtained, the reconstruction operation cannot be performed accurately, and as a result, it is difficult to generate correct reconstructed image data.
Therefore, in order to generate correct reconstructed image data, it is necessary to accurately determine the parameters (A) to (J). For this purpose, the main lens 21, the microlens array 22, and the image pickup device 23 need to be determined. Ideally, the alignment should be accurate. However, a certain amount of manufacturing error or mounting error occurs during the manufacturing of the image pickup apparatus 2, but it is very difficult to perform alignment so as to completely eliminate these errors.
Therefore, the camera manufacturing apparatus 1 of the present embodiment executes a process for obtaining these errors (hereinafter referred to as “calibration process”), and causes the imaging apparatus 2 to record information such as the errors as initial data.
The imaging device 2 can obtain the values of the above parameters (A) to (J) almost accurately by appropriately correcting using the initial data when performing reconstruction calculation after shipment, As a result, accurate reconstructed image data can be generated.

  Further, the calibration process executed by the camera manufacturing apparatus 1 of the present embodiment will be described in further detail below.

Among the parameters (A) to (J) described above, parameters (F) to (J) are likely to cause errors during manufacturing.
Here, in the optical system of the imaging device 2, the direction parallel to the optical axis of the main lens 21 is the z axis, and the upper axis of the imaging element 23 in the direction perpendicular to the optical axis is the x axis. Of the directions perpendicular to the axis, the right axis of the image sensor 23 is defined as the y axis.
Under such a definition, among the parameters (F) to (J), the error in the z direction in the parameter (G) is small enough to be ignored compared to the distance between the main lens 21 and the microlens array 22. Suppose there is.
Regarding parameters (I) and (J), it is assumed that the main lens 21 and the image sensor 23 are attached in parallel, and attention is paid only to the parameter (J).
Further, the parameter (H) is considered separately for the xy direction perpendicular to the optical axis of the main lens 21 and the z direction parallel to the optical axis.

From the above, it is assumed that the following errors (a) to (d) are calculated in the calibration process of the present embodiment.
(A) Error in the xy direction of the micro lens array 22 (b) Error in the z direction of the micro lens array 22 (c) Pitch error of each micro lens 22-i (error in the xy direction of each micro lens 22-i)
(D) Parallelism error between the microlens array 22 and the image sensor 23 (z direction error of each microlens 22-i)

  FIG. 3 is a functional block diagram showing a functional configuration example of the control device 12 of FIG. 1 for exhibiting the execution function of the calibration process for calculating such errors (a) to (d).

  The control device 12 includes an operation unit 31, a communication unit 32, and a calibration unit 33.

The operation unit 31 includes various buttons and receives an instruction operation from the operator.
The communication unit 32 controls communication performed with the imaging device 2. The communication unit 32 also controls communication with other devices (not shown) via the Internet or the like.
The calibration unit 33 includes an image acquisition unit 41, an ideal parameter acquisition unit 42, an error calculation unit 43, and an error information output unit 44.

  The image acquisition unit 41 executes control for causing the imaging device 2 to image the light source 11 that is radiating parallel light or uniform diffused light via the communication unit 32, and outputs light field image data obtained as a result of the control. Then, it is acquired from the imaging device 2 as calibration image data.

The ideal parameter acquisition unit 42 is obtained in advance as a result of imaging the light source 11 irradiating parallel light or uniform diffused light in advance by an ideal imaging device (prepared separately from the imaging device 2). A parameter obtained from data of a typical light field image or each value (may be a design value) of an optical system of an ideal imaging device is acquired as an ideal parameter.
As each value of the optical system of the ideal image pickup apparatus, ideal values of the parameters (A) to (J) described above, or various values that can be calculated by a predetermined calculation using these ideal values. Can be adopted.
A specific example of the ideal parameter will be described later with reference to FIGS.

The error calculation unit 43 calculates the errors (a) to (d) described above based on the calibration image data acquired by the image acquisition unit 41 and the ideal parameters acquired by the ideal parameter acquisition unit 42. .
A specific method for calculating the errors (a) to (d) will be described later with reference to FIGS.

The error information output unit 44 includes information indicating the errors (a) to (d) calculated by the error calculation unit 43 or information generated using at least a part of these errors (a) to (d). The error information is output to the imaging device 2 via the communication unit 32.
The imaging device 2 stores the error information in an internal memory, and uses it as correction information as appropriate when calculating reconstruction.

Next, the flow of the calibration process among the processes executed by the camera manufacturing apparatus 1 including the control device 12 of FIG. 3 will be described.
FIG. 4 is a flowchart illustrating an example of the flow of calibration processing executed by the imaging device manufacturing apparatus of FIG.

  In the calibration process, for example, each power source of the light source 11 and the control device 12 constituting the camera manufacturing apparatus 1 is turned on, and the assembled imaging device 2 is arranged at a predetermined position with respect to the light source 11 and is controlled. The process is started when communication with the device 12 is possible.

In step S1, the camera manufacturing apparatus 1 executes a preparation process.
Specifically, as one of the preparation processes, the camera manufacturing apparatus 1 selects a parallel light irradiation device as the light source 11, and uses the optical axis of the parallel light irradiation device as the optical axis of the main lens 21 of the imaging device 2. Match.
In the present embodiment, the camera manufacturing apparatus 1 is provided with a moving mechanism (not shown) that moves at least one of the light source 11 and the imaging apparatus 2. When the moving mechanism moves at least one of the light source 11 and the imaging device 2, the optical axes of the light source 11 and the main lens 21 coincide with each other. However, the operator of the camera manufacturing apparatus 1 may cause the optical axes of the light source 11 and the main lens 21 to coincide with each other by manually moving at least one of the light source 11 and the imaging device 2.
Moreover, the camera manufacturing apparatus 1 irradiates the imaging device 2 with parallel light from a parallel light irradiation device as the light source 11 as one of the preparation processes.

In step S <b> 2, the image acquisition unit 41 of the control device 12 acquires light field image data for the parallel light source 11.
That is, the image acquisition unit 41 causes the imaging device 2 to image the light source 11 that emits the parallel light via the communication unit 32, and uses the resulting light field image data as calibration image data. Get from 2.

In step S <b> 3, the error calculation unit 43 of the control device 12 calculates the error (a), that is, the error in the xy direction of the microlens array 22.
Specifically, the error calculation unit 43 calculates, as an error (a), an error with respect to the center of the ideal light field image with respect to the center of the sub-image of the calibration image acquired as data in the process of step S2. To do.
Further details of the error (a) calculation process will be described later with reference to FIG.

In step S4, the error calculation unit 43 of the control device 12 calculates the error (b), that is, the error in the z direction of the microlens array 22.
Specifically, the error calculation unit 43 calculates the diameter of the sub image in the calibration image acquired as data in the process of step S2. Next, the error calculation unit 43 calculates a deviation of the distance between the microlens array 22 and the image sensor 23 from the focal length of the microlens 22-i as an error (b) based on the diameter.
Further details of the error (b) calculation process will be described later with reference to FIG.

  In step S <b> 5, the camera manufacturing apparatus 1 switches to the uniform diffused light irradiation apparatus as the light source 11 and irradiates the imaging apparatus 2 with the uniform diffused light.

In step S <b> 6, the image acquisition unit 41 of the control device 12 acquires light field image data for the uniformly diffused light source 11.
That is, the image acquisition unit 41 causes the imaging device 2 to image the uniform diffused light irradiation device that emits the uniform diffused light via the communication unit 32, and obtains the data of the light field image obtained as a result of the calibration image. It is acquired from the imaging device 2 as data.

In step S7, the error calculation unit 43 of the control device 12 calculates the error (c), that is, the pitch error of each microlens 22-i (the xy direction error of each microlens 22-i).
Specifically, the error calculation unit 43 compares the center position of each sub-image in the calibration image acquired as data in the process of step S6 with respect to the center position of each sub-image in the ideal light field image. The deviation is determined as error (c).
That is, the error calculation unit 43 compares the center positions of the sub-images of the calibration image and the ideal light field image. Based on the result of the comparison, the error calculation unit 43 calculates a deviation of the arrangement of the individual microlenses 22-i from the ideal arrangement as an error (c).
Further details of the error (c) calculation process will be described later with reference to FIG.

In step S8, the error calculation unit 43 of the control device 12 calculates an error (d), that is, a parallelism error between the microlens array 22 and the image sensor 23 (z direction error of each microlens 22-i).
Specifically, the error calculation unit 43 calculates the error of the distance between each micro lens 22-i and the image sensor 23 as an error (d).
That is, the error calculation unit 43 calculates each diameter of each sub-image in the calibration image acquired as data in the process of step S6. Based on each of these diameters, the error calculation unit 43 uses, as an error (d), a deviation from the focal length of the microlens 22-i with respect to the distance between each microlens 22-i and the image sensor 23. calculate.
Further details of the error (d) calculation process will be described later with reference to FIG.

  In step S9, the error information output unit 44 of the control device 12 includes information indicating the errors (a) to (d) calculated by the error calculation unit 43, or at least a part of these errors (a) to (d). The information generated using is output to the imaging apparatus 2 via the communication unit 32 as error information.

This completes the calibration process.
Note that the imaging device 2 stores error information in an internal memory and uses it appropriately when performing reconstruction.

  Next, with reference to FIGS. 5 to 8, examples of specific calculation methods for each of the errors (a) to (d) obtained in the calibration process will be described.

FIG. 5 is a diagram for explaining an example of a method for calculating the error (a) obtained in the calibration process, that is, the error in the xy direction of the microlens array 22.
Specifically, in FIG. 5A, the optical system of the imaging apparatus 2 in the case where the error (a) does not occur, that is, in an ideal state, is shown on the left side, and on the right side, An example of an ideal light field image (to be precise, a binarized image described later) captured by the imaging device 2 is shown.
In FIG. 5 (b), the optical system of the imaging device 2 when the error (a) occurs is shown on the left side, and the light field image captured by the imaging device 2, that is, calibration is shown on the right side. An example of a motion image (to be precise, a binarized image described later) is shown.

As described above, the error (a) is executed as the process of step S3 in the calibration process of FIG.
Specifically, for example, the error (a) is calculated according to the following steps S3-1 to S3-4.

That is, in step S3-1, the error calculation unit 43 converts the light field image (calibration image) data acquired in the process of step S2 into binarized image data.
The binarized image data is a pixel value that satisfies a predetermined condition (for example, a threshold value or more) in the light field image data, the pixel value is converted to “1” (converted to a white pixel), and the others The pixel value (for example, a pixel value less than a threshold value) is a pixel value converted to “0” (converted to a black pixel).

  In step S <b> 3-2, the error calculation unit 43 extracts a white area (a pixel area having a pixel value of 1) as a sub-image area using the binarized image data.

  In step S3-3, the error calculation unit 43 calculates the center of gravity of the extracted sub-image region, and sets the center of gravity as the center position (x, y) of the sub-image of the calibration image.

In step S3-4, the error calculation unit 43 calculates an error with respect to the center position (xc, yc) of the ideal light field image with respect to the center position (x, y) of the sub-image of the calibration image. ).
That is, if the error (a) is described as (ArrErrorX, ArrErrorY), the following expressions (1) and (2) are calculated in the process of step S3-4.
ArrErrorX = x − xc (1)
ArrErrorY = y − yc (2)
Note that an ideal light field image is captured in advance as shown in FIG. As a result, the center position (xc, yc) of the ideal light field image is also registered in advance as one of the ideal parameters, and is acquired by the ideal parameter acquisition unit 42 before the process of step S3-4.

FIG. 6 is a diagram for explaining an example of a method for calculating the error (b) obtained in the calibration process, that is, the error in the z direction of the microlens array 22.
Specifically, in FIG. 6A, the optical system of the imaging apparatus 2 in the case where the error (b) does not occur, that is, in an ideal state, is shown on the left side. An example of an ideal light field image (to be precise, a binarized image described later) captured by the imaging device 2 is shown.
In FIG. 6B, the optical system of the imaging device 2 when the error (b) occurs is shown on the left side, and the light field image captured by the imaging device 2, that is, the calibration is shown on the right side. An example of a motion image (to be precise, a binarized image described later) is shown.

As described above, the error (b) is executed as the process of step S4 in the calibration process of FIG.
Specifically, for example, the error (b) is calculated according to the following steps S4-1 to S4-3.

  That is, in step S4-1, the error calculation unit 43 determines the center position of the sub-image area extracted in the process of step S3-2 and the sub-image of the calibration image calculated in the process of step S3-3. Get (x, y).

  In step S4-2, the error calculation unit 43 calculates the diameter Dsub of the sub-image of the calibration image based on the acquired sub-image area and the center position (x, y) of the sub-image of the calibration image (FIG. 6B). )) Is calculated.

In step S4-3, the error calculation unit 43 obtains the error (b) from the following equation (3).
Dsub = DML * (fμL + ArrErrorZ) / fML (3)
In Expression (3), as shown in FIG. 6B, DML is the diameter of the main lens 21, fμL is the focal length of the microlens 22-i, and ArrErrorZ is the error (b), that is, the microlens array 22. The error in the z direction and fML indicate the focal length of the main lens 21, respectively.
Each of the diameter DML of the main lens 21, the focal length fμL of the microlens 22-i, and the focal length fML of the main lens 21 is registered in advance as one of the ideal parameters, and the process of step S4-3 is performed. It is acquired by the ideal parameter acquisition unit 42 before.

FIG. 7 is a diagram for explaining an example of a calculation method of the error (c) obtained by the calibration process, that is, the pitch error of each microlens 22-i (the xy direction error of each microlens 22-i).
Specifically, in FIG. 7A, the optical system of the imaging apparatus 2 in the case where the error (c) does not occur, that is, in an ideal state is shown on the left side, and on the right side, An example of an ideal light field image (to be precise, a binarized image described later) captured by the imaging device 2 is shown.
In FIG. 7B, the optical system of the imaging device 2 when the error (c) occurs is shown on the left side, and the light field image captured by the imaging device 2, that is, calibration is shown on the right side. An example of a motion image (to be precise, a binarized image described later) is shown.

As described above, the error (c) is executed as the process of step S7 in the calibration process of FIG.
Specifically, for example, the error (c) is calculated according to the following steps S7-1 to S7-4.

  That is, in step S7-1, the error calculation unit 43 converts the light field image (calibration image) data acquired in the process of step S6 into binary image data.

  In step S <b> 7-2, the error calculation unit 43 extracts each white area (a pixel area having a pixel value of 1) as each sub-image area using the binarized image data.

  In step S <b> 7-3, the error calculation unit 43 calculates the centroid for each extracted sub-image region, and sets each centroid as the center position (xi, yi) of each sub-image. .

In step S7-4, the error calculator 43 determines the center position (xbasei, ybasei) of each sub-image of the ideal light field image for each center position (xi, yi) of each sub-image of the calibration image. Is calculated as error (c).
That is, if the error (c) is described as (LensErrorXi, LensErrorYi), the following expressions (4) and (5) are calculated in the process of step S7-4.
LensErrorXi = xi − xbasei (4)
LensErrorYi = yi − ybasei (5)
Note that an ideal light field image is captured in advance as shown in FIG. As a result, the center position (xbasei, ybasei) of each sub-image of the ideal light field image is also registered in advance as one of the ideal parameters, and the ideal parameter acquisition unit before the processing of step S7-4 42.

FIG. 8 is a diagram for explaining an example of a method for calculating an error (d) obtained in the calibration process, that is, a parallelism error between the microlens array 22 and the image sensor 23 (z direction error of each microlens 22-i). is there.
Specifically, in FIG. 8A, the optical system of the imaging apparatus 2 in the case where the error (d) does not occur, that is, in an ideal state, is shown on the left side, and on the right side, An example of an ideal light field image (to be precise, a binarized image described later) captured by the imaging device 2 is shown.
In FIG. 8B, the optical system of the imaging apparatus 2 when the error (d) occurs is shown on the left side, and the light field image captured by the imaging apparatus 2 on the right side, that is, calibration. An example of a motion image (to be precise, a binarized image described later) is shown.

As described above, the error (d) is executed as the process of step S8 in the calibration process of FIG.
Specifically, for example, the error (d) is calculated according to the following steps S8-1 to S8-3.

  That is, in step S8-1, the error calculation unit 43 determines the area of each sub-image extracted in the process of step S7-2 and the center position of each sub-image calculated in the process of step S7-3 ( xi, yi).

  In step S8-2, the error calculation unit 43 sets the corresponding sub-image diameter Dsubi (FIG. 8) for each set of the sub-image area and the center position (xi, yi) of the corresponding sub-image. (See (b)).

In step S8-3, the error calculation unit 43 obtains an error (d) for each sub-image from the following equation (6).
Dsubi = DML * (fμL + zi) / fML (6)
In Expression (6), as shown in FIG. 8B, DML is the diameter of the main lens 21, fμL is the focal length of the microlens 22-i, and zi is an error (d for each microlens 22-i. ), I.e., the z-direction error of each microlens 22-i, and fML indicates the focal length of the main lens 21.
As described above, each of the diameter DML of the main lens 21, the focal length fμL of the microlens 22-i, and the focal length fML of the main lens 21 is registered in advance as one of the ideal parameters. It is acquired by the ideal parameter acquisition unit 42 before the process of S8-3.

As described above, the camera manufacturing apparatus 1 according to this embodiment manufactures the imaging apparatus 2 including the main lens 21, the microlens array 22 including the plurality of microlenses 22-i, and the imaging element 23 in the optical system. .
The camera manufacturing apparatus 1 includes a light source 11 that irradiates the imaging apparatus 2 with parallel light or uniform diffused light, and a control device 12. The control device 12 includes an image acquisition unit 41, an ideal parameter acquisition unit 42, an error calculation unit 43, and an error information output unit 44.
The image acquisition unit 41 is a set of a plurality of sub-images generated by each of the plurality of microlenses 22-i obtained as a result of the light source 11 irradiating parallel light or uniform diffused light being imaged by the imaging device 2. Data of a light field image composed of a body is acquired as calibration image data.
The ideal parameter acquisition unit 42 is obtained in advance as a result of imaging the light source 11 irradiating parallel light or uniform diffused light in advance by an ideal imaging device (prepared separately from the imaging device 2 to be manufactured). An ideal light field image data or a parameter obtained from an ideal design value of the optical system of the imaging apparatus is acquired as an ideal parameter.
The error calculation unit 43 uses the calibration image data acquired by the image acquisition unit 41 and the ideal parameter acquired by the ideal parameter acquisition unit 42 to generate the position of the microlens array 22 that is generated when the imaging device 2 is manufactured. And errors in parallelism, that is, the above errors (a), (b), and (d) are calculated.
The error calculation unit 43 further calculates an error of each pitch of the plurality of microlenses 22-i, that is, the error (c) described above, using the calibration image data and the ideal parameters.
The error information output unit 44 outputs information indicating the errors (a) to (d) calculated by the error calculation unit 43 or information generated based on these errors to the imaging device 2 as error information ( (See step S9 in FIG. 4 above).

FIG. 9 is a block diagram showing a hardware configuration of the control device 12 when the above-described series of processing is executed by software.
The control device 12 is composed of a personal computer.

  In addition to the operation unit 31 and the communication unit 32 described above, the control device 12 includes a CPU 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a bus 104, an input / output interface 105, a display Unit 106, storage unit 107, and drive 108.

The CPU 101 executes various processes according to a program recorded in the ROM 102 or a program loaded from the storage unit 107 to the RAM 103.
The RAM 103 also appropriately stores data necessary for the CPU 101 to execute various processes.

  For example, the calibration unit 33 of FIG. 3 described above can be configured as a combination of hardware called the CPU 101 and a program (software) stored in the ROM 102 or the like.

  The CPU 101, ROM 102, and RAM 103 are connected to each other via a bus 104. An input / output interface 105 is also connected to the bus 104. In addition to the operation unit 31 and the communication unit 32 described above, a display unit 106, a storage unit 107, and a drive 108 are connected to the input / output interface 105.

  The display unit 106 is configured by a liquid crystal display or the like, and displays a GUI (Graphical User Interface) image or the like operated by the operator during the above-described calibration processing.

  The storage unit 107 is composed of a hard disk or the like, and stores data of various parameters. The storage unit 107 also stores various data necessary for various types of information processing, such as image data, various flag values, threshold values, and the like.

  A removable medium 111 made of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately attached to the drive 108. The computer program read from the removable medium 111 by the drive 108 is installed in the storage unit 107 or the like as necessary.

  The imaging device 2 records this error information as one of the initial data in the built-in memory. Thereafter, the imaging device 2 is subjected to various processes necessary for commercialization and shipped.

  FIG. 10 is a block diagram showing a hardware configuration of the imaging apparatus according to the present invention after being manufactured and shipped by the imaging apparatus manufacturing apparatus.

  The imaging apparatus 2 includes a CPU 201, a ROM 202, a RAM 203, a bus 204, an input / output interface 205, an imaging unit 206, an input unit 207, an output unit 208, a storage unit 209, a communication unit 210, and a drive. 211.

The CPU 201 executes various processes according to a program recorded in the ROM 202 or a program loaded from the storage unit 209 to the RAM 203.
The ROM 202 also stores error information output from the imaging device manufacturing apparatus 1.

  The RAM 203 appropriately stores data necessary for the CPU 201 to execute various processes.

  The CPU 201, ROM 202, and RAM 203 are connected to each other via a bus 204. An input / output interface 205 is also connected to the bus 204. An imaging unit 206, an input unit 207, an output unit 208, a storage unit 209, a communication unit 210, and a drive 211 are connected to the input / output interface 205.

  The imaging unit 206 includes a main lens 21, a microlens array 22, and an imaging element 23. Further details of the imaging unit 206 will be described later with reference to FIG.

The input unit 207 is configured with various buttons such as a shutter button (not shown), and inputs various information according to a user's instruction operation.
The output unit 208 includes a monitor, a speaker, and the like, and outputs various images and various sounds.
The storage unit 209 is configured by a hard disk, a DRAM (Dynamic Random Memory), or the like, and stores data of various images such as a light field image and a reconstructed image.
The communication unit 210 controls communication performed with other devices (not shown) via a network including the Internet.

  A removable medium 221 composed of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately attached to the drive 211. The program read from the removable medium 221 by the drive 211 is installed in the storage unit 209 as necessary. The removable medium 221 can also store various data such as image data stored in the storage unit 209 in the same manner as the storage unit 209.

  FIG. 11 is a schematic diagram illustrating a configuration example of an optical system in the imaging apparatus having the configuration illustrated in FIG.

In the optical system of the imaging device 2, the main lens 21, the microlens array 22, and the imaging element 23 are arranged in that order when viewed from the object plane ob that is a subject.
In the microlens array 22, each of N (N is an arbitrary integer value of 2 or more) microlenses 22-1 to 22-N is regularly and continuously arranged.

  The main lens 21 condenses the light beam emitted from the light source, forms an image on a predetermined surface Ma, and enters the microlens array 22. Hereinafter, the surface Ma imaged by the main lens 21 is referred to as a “main lens imaging surface Ma”.

The microlenses 22-i (i is an integer value in the range of 1 to N) in the microlens array 22 condenses light beams incident from the object plane ob via the main lens 21 in each incident direction. Thus, a sub-image is formed on the image sensor 23.
That is, in the imaging device 23, a plurality of sub-images are formed by each of the plurality of microlenses 22-1 to 22-N, and a light field image that is an aggregate of the plurality of sub-images is generated.

Here, consider a case where the imaging device 2 generates reconstructed image data from light field image data obtained as a result of imaging the object plane ob.
In this case, when the imaging device 2 sets one point on a surface at an arbitrary distance as a point of interest, light from the point of interest passes through the main lens 21 and the microlens array 22 and any pixel in the image sensor 23. To be distributed. Hereinafter, the surface to be reconstructed, that is, the surface on which the attention point is set is referred to as a “reconstruction surface”. Then, the imaging device 2 corresponds to the distributed pixels in the light field image data. By integrating the pixel value, the pixel value of the pixel corresponding to the target point in the reconstructed image is estimated and calculated.
The imaging device 2 generates data of the reconstructed image by executing such estimation calculation for each pixel of the reconstructed image.
The processing until the imaging device 2 generates the reconstructed image data in this way is hereinafter referred to as “reconstruction processing”.

FIG. 12 is a flowchart for explaining the flow of reconstruction processing executed by the imaging apparatus of FIG.
It is assumed that the imaging device 2 images a subject and stores light field image data obtained as a result in the storage unit 209 or the like before reconstruction processing. Further, it is assumed that error information output from the imaging device manufacturing apparatus of FIG. 1 is stored in the ROM 202 when the imaging device 2 is shipped.

In step S21, the CPU 201 of the imaging apparatus 2 acquires light field image data from the storage unit 209 or the like.
In step S22, the CPU 201 acquires error information from the ROM 202 as correction information.
In step S <b> 23, the CPU 201 sets a surface at a predetermined distance in front of the main lens 21 of the imaging device 2 as a reconstruction surface.

In step S24, the CPU 201 sets one point on the reconstruction plane as a reconstruction target pixel.
In step S25, the CPU 201 calculates the distribution pixel range in consideration of the correction information acquired in the process of step S22. The distribution pixel range is a pixel range in the image sensor 23 to which light from the reconstructed pixel of interest is distributed via the main lens 21 and the microlens array 22, that is, a pixel range in the light field image. That is, by obtaining the distribution pixel range in consideration of the correction information, the values of the parameters (A) to (J) described above are obtained almost accurately, and as a result, an appropriate distribution pixel range is calculated.
In step S26, the CPU 201 integrates the pixel values of the respective pixels within the distribution pixel range.
In step S27, the CPU 201 sets the integral value obtained as a result of the process in step S26 to the pixel value of the reconstructed pixel of interest.

In step S <b> 28, the CPU 201 determines whether all points on the reconstruction plane have been set as reconstruction target pixels.
If there is a point that has not yet been set as the reconstructed target pixel among the points on the reconstructed surface, it is determined as NO in step S28, the process returns to step S24, and the subsequent processes are repeated. . That is, each point on the reconstruction plane is sequentially set as a reconstructed target pixel, and each time the loop processing of steps S24 to S28 is repeatedly performed, the pixel value of the reconstructed target pixel is set.
In this manner, the pixel value of each pixel corresponding to each point on the reconstruction plane is set, thereby generating reconstruction image data. Thereby, it determines with it being YES in step S28, and a process progresses to step S29.
In step S <b> 29, the CPU 201 displays and outputs the reconstructed image from the output unit 208.
As a result, the reconstruction process ends.
In this way, when the image capturing apparatus 2 generates reconstructed image data by reconstruction processing, the error information is read from the built-in memory and used as correction information.
This makes it possible to easily reconstruct accurate refocused image data from light field image data.
In other words, when reconstructing the refocused image data, manufacturing errors and mounting errors that are different for each imaging device 2 are absorbed, so that the optical system of the imaging device 2 can be positioned more accurately than necessary. Unnecessary, that is, accuracy errors during manufacturing can be tolerated to some extent. As a result, it is possible to manufacture the imaging device 2 easily and in a short time, and at low cost.

  In addition, this invention is not limited to the above-mentioned embodiment, The deformation | transformation in the range which can achieve the objective of this invention, improvement, etc. are included in this invention.

For example, in the above-described embodiment, the error calculation unit 43 calculates a combination of the errors (a) to (d). However, the error calculation unit 43 is not particularly limited to these combinations, and any error among the errors (a) to (d). For example, a combination of errors (a), (b), and (d) may be calculated.
In this case, even if only the combination of the errors (a), (b), and (d) is employed, it is possible to achieve an effect that it is possible to easily reconstruct accurate refocused image data. . However, the effect becomes more remarkable when all of the errors (a) to (d) are adopted as in the present embodiment.
Alternatively, the error calculation unit 43 is an error other than the errors (a) to (d), and causes the error due to the parameters (A) to (J) described above together with the errors (a) to (d) or those It may be calculated separately.

Further, for example, in the above-described embodiment, the calibration process is performed by the imaging device manufacturing apparatus disposed in a factory or the like before the imaging device 2 is shipped, but the above-described light source 11 may be prepared. If possible, it may be executed after the imaging device 2 is shipped.
In this case, the error calculation unit 43 can also calculate an error associated with the use of the imaging device 2 after shipment. As a result, it is possible to easily reconstruct accurate refocused image data from light field image data even if the imaging device 2 has been used for many years.
In this case, it is sufficient that the control device 12 has a configuration capable of communicating with the imaging device 2 and capable of executing the above-described calibration process. In addition to the personal computer shown in FIG. Etc. can be widely applied.
Furthermore, the imaging device 2 itself may be provided with the function of the control device 12.

  By the way, the above-described series of processes can be executed by hardware or can be executed by software.

  When a series of processing is executed by software, a program constituting the software is installed on a computer or the like from a network or a recording medium. The computer may be a computer incorporated in dedicated hardware. The computer may be a computer capable of executing various functions by installing various programs, for example, a general-purpose personal computer.

  The recording medium including such a program is not only configured by the removable medium 111 in FIG. 9 or the removable medium 221 in FIG. 10 distributed separately from the apparatus main body in order to provide the program to the user, but also in the apparatus main body. It is composed of a recording medium or the like provided to the user in a preinstalled state. The removable medium is composed of, for example, a magnetic disk (including a floppy disk), an optical disk, a magneto-optical disk, or the like. The optical disk is composed of, for example, a CD-ROM (Compact Disk-Read Only Memory), a DVD (Digital Versatile Disk), or the like. The magneto-optical disk is configured by an MD (Mini-Disk) or the like. The recording medium provided to the user in a state of being incorporated in the apparatus main body in advance is, for example, the ROM 102 in FIG. 9 or the ROM 202 in FIG. 10 in which the program is recorded, the storage unit 107 in FIG. 9 or the storage in FIG. The hard disk included in the unit 209 is configured.

  In the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in time series along the order, but is not necessarily performed in time series, either in parallel or individually. The process to be executed is also included.

  As mentioned above, although several embodiment of this invention was described, these embodiment is only an illustration and does not limit the technical scope of this invention. The present invention can take other various embodiments, and various modifications such as omission and replacement can be made without departing from the gist of the present invention. These embodiments and modifications thereof are included in the scope and gist of the invention described in this specification and the like, and are included in the invention described in the claims and the equivalents thereof.

The invention described in the scope of claims at the beginning of the filing of the present application will be appended.
[Appendix 1]
An apparatus for manufacturing an imaging apparatus including a main lens, a microlens array including a plurality of microlenses, and an imaging element in an optical system,
A light source for irradiating the imaging device with parallel light or uniform diffused light;
Light field image data consisting of a collection of a plurality of sub-images generated by each of the plurality of microlenses, obtained as a result of the light source emitting parallel light or uniform diffused light being imaged by the imaging device An image acquisition unit for acquiring
Data of an ideal light field image obtained in advance as a result of the light source irradiating parallel light or uniform diffused light being previously imaged by an ideal imaging device, or the ideal imaging device An ideal parameter acquisition unit for acquiring, as an ideal parameter, a parameter obtained from the design value of the optical system;
Using the data of the light field image acquired by the image acquisition unit and the ideal parameter acquired by the ideal parameter acquisition unit, the position and parallelism of the microlens array generated at the time of manufacturing the imaging device An error calculation unit for calculating each error;
An error information output unit that outputs information indicating each error calculated by the error calculation unit or information generated based on each error to the imaging device as error information;
An apparatus for manufacturing an imaging apparatus, comprising:
[Appendix 2]
The error calculation unit further calculates an error of each pitch of the plurality of microlenses using the data of the light field image and the ideal parameter.
The imaging apparatus manufacturing apparatus according to appendix 1.
[Appendix 3]
An apparatus for manufacturing an imaging apparatus including a main lens, a microlens array including a plurality of microlenses, and an imaging element in an optical system,
In a manufacturing method executed by a manufacturing apparatus for an imaging apparatus including a light source that irradiates the imaging apparatus with parallel light or uniform diffused light,
An imaging step of causing the imaging device to image the light source in a state where the imaging device is irradiated with parallel light or uniform diffused light,
An image acquisition step of acquiring light field image data composed of a collection of a plurality of sub-images generated by each of the plurality of microlenses obtained as a result of imaging by the imaging device in the imaging step;
Ideal light field image data obtained in advance as a result of pre-imaging the light source irradiating the parallel light or uniform diffused light by an ideal imaging device, or of the ideal imaging device An ideal parameter acquisition step of acquiring a parameter obtained from the design value of the optical system as an ideal parameter;
Using the data of the light field image acquired in the image acquisition step and the ideal parameter acquired in the ideal parameter acquisition step, the position and parallelism of the microlens array generated at the time of manufacturing the imaging device An error calculating step for calculating each error;
An error information output step of outputting information indicating each error calculated in the error calculation step or information generated based on each error to the imaging apparatus as error information;
A method for manufacturing an imaging apparatus, comprising:
[Appendix 4]
An apparatus for manufacturing an imaging apparatus including a main lens, a microlens array composed of a plurality of microlenses, and an imaging element in an optical system, the imaging apparatus including a light source that irradiates the imaging apparatus with parallel light or uniform diffused light To the computer that controls the device,
Light field image data consisting of a collection of a plurality of sub-images generated by each of the plurality of microlenses, obtained as a result of the light source emitting parallel light or uniform diffused light being imaged by the imaging device Image acquisition function,
Data of an ideal light field image obtained in advance as a result of the light source irradiating parallel light or uniform diffused light being previously imaged by an ideal imaging device, or the ideal imaging device Ideal parameter acquisition function that acquires the parameters obtained from the design values of the optical system as ideal parameters,
Using the data of the light field image acquired by the image acquisition function and the ideal parameter acquired by the ideal parameter acquisition function, the position and parallelism of the microlens array generated at the time of manufacturing the imaging device Error calculation function to calculate each error,
An error information output function for outputting, as error information, information indicating each error calculated by the error calculation function or information generated based on each error to the imaging apparatus;
A program characterized by realizing.
[Supplementary Note 5] An optical system including a main lens, a microlens array composed of a plurality of microlenses, and an imaging device, images a subject by the optical system, and outputs light field image data obtained as a result. Imaging means;
Storage means for preliminarily storing error information indicating each error in the position and parallelism of the microlens array generated at the time of manufacture;
Control means for executing control based on the light field image data output from the image pickup means to generate reconstructed image data that is in focus at a position different from the time when the light field image was picked up. When,
With
The control means generates data of the reconstructed image using the error information stored in the storage means as correction information.
Imaging device.

DESCRIPTION OF SYMBOLS 1 ... Camera manufacturing apparatus, 2 ... Imaging device, 11 ... Light source, 12 ... Control apparatus, 21 ... Main lens, 22 ... Micro lens array, 22-1 thru | or 22-N ... Microlens, 23 ... Image sensor, 31 ... Operation part, 32 ... Communication part, 33 ... Calibration part, 41 ... Image acquisition part, 42 ... Acquisition of ideal parameters Unit, 43 ... error calculation unit, 44 ... error information output unit, 101 ... CPU

Claims (5)

An apparatus for manufacturing an imaging apparatus including a main lens, a microlens array including a plurality of microlenses, and an imaging element in an optical system,
A light source for irradiating the imaging device with parallel light or uniform diffused light;
Light field image data consisting of a collection of a plurality of sub-images generated by each of the plurality of microlenses, obtained as a result of the light source emitting parallel light or uniform diffused light being imaged by the imaging device An image acquisition unit for acquiring
Data of an ideal light field image obtained in advance as a result of the light source irradiating parallel light or uniform diffused light being previously imaged by an ideal imaging device, or the ideal imaging device An ideal parameter acquisition unit for acquiring, as an ideal parameter, a parameter obtained from the design value of the optical system;
Using the data of the light field image acquired by the image acquisition unit and the ideal parameter acquired by the ideal parameter acquisition unit, the position and parallelism of the microlens array generated at the time of manufacturing the imaging device An error calculation unit for calculating each error;
An error information output unit that outputs information indicating each error calculated by the error calculation unit or information generated based on each error to the imaging device as error information;
An apparatus for manufacturing an imaging apparatus, comprising: The error calculation unit further calculates an error of each pitch of the plurality of microlenses using the data of the light field image and the ideal parameter.
The imaging device manufacturing apparatus according to claim 1. An apparatus for manufacturing an imaging apparatus including a main lens, a microlens array including a plurality of microlenses, and an imaging element in an optical system,
In a manufacturing method executed by a manufacturing apparatus for an imaging apparatus including a light source that irradiates the imaging apparatus with parallel light or uniform diffused light,
An imaging step of causing the imaging device to image the light source in a state where the imaging device is irradiated with parallel light or uniform diffused light,
An image acquisition step of acquiring light field image data composed of a collection of a plurality of sub-images generated by each of the plurality of microlenses obtained as a result of imaging by the imaging device in the imaging step;
Ideal light field image data obtained in advance as a result of pre-imaging the light source irradiating the parallel light or uniform diffused light by an ideal imaging device, or of the ideal imaging device An ideal parameter acquisition step of acquiring a parameter obtained from the design value of the optical system as an ideal parameter;
Using the data of the light field image acquired in the image acquisition step and the ideal parameter acquired in the ideal parameter acquisition step, the position and parallelism of the microlens array generated at the time of manufacturing the imaging device An error calculating step for calculating each error;
An error information output step of outputting information indicating each error calculated in the error calculation step or information generated based on each error to the imaging apparatus as error information;
A method for manufacturing an imaging apparatus, comprising: An apparatus for manufacturing an imaging apparatus including a main lens, a microlens array composed of a plurality of microlenses, and an imaging element in an optical system, the imaging apparatus including a light source that irradiates the imaging apparatus with parallel light or uniform diffused light To the computer that controls the device,
Light field image data consisting of a collection of a plurality of sub-images generated by each of the plurality of microlenses, obtained as a result of the light source emitting parallel light or uniform diffused light being imaged by the imaging device Image acquisition function,
Data of an ideal light field image obtained in advance as a result of the light source irradiating parallel light or uniform diffused light being previously imaged by an ideal imaging device, or the ideal imaging device Ideal parameter acquisition function that acquires the parameters obtained from the design values of the optical system as ideal parameters,
Using the data of the light field image acquired by the image acquisition function and the ideal parameter acquired by the ideal parameter acquisition function, the position and parallelism of the microlens array generated at the time of manufacturing the imaging device Error calculation function to calculate each error,
An error information output function for outputting, as error information, information indicating each error calculated by the error calculation function or information generated based on each error to the imaging apparatus;
A program characterized by realizing. An imaging unit having an optical system including a main lens, a microlens array including a plurality of microlenses, and an imaging device, imaging a subject by the optical system, and outputting data of a light field image obtained as a result;
Storage means for preliminarily storing error information indicating each error in the position and parallelism of the microlens array generated at the time of manufacture;
Control means for executing control based on the light field image data output from the image pickup means to generate reconstructed image data that is in focus at a position different from the time when the light field image was picked up. When,
With
The control means generates data of the reconstructed image using the error information stored in the storage means as correction information.
Imaging device.