JP2009290268A - Imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging apparatus for increasing the light receiving region of an imaging pixel. <P>SOLUTION: The aperture diaphragm 10 of an imaging lens 11 includes a rectangular opening 10A. Thus, an image (light receiving image 13D-1) received on an imaging element 13 is also turned to a rectangular shape. Thus, compared to the case of using a conventional aperture diaphragm having a circular opening, it becomes easier to form the respective light receiving images 13D-1 densely without a gap in the imaging element 13. Thus, the light receiving region of the imaging pixel is increased. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an imaging device using an imaging lens and an imaging element.

  Conventionally, various imaging devices have been proposed and developed. There has also been proposed an imaging apparatus that performs predetermined image processing on imaging data obtained by imaging and outputs the data.

For example, Patent Document 1 and Non-Patent Document 1 propose an imaging apparatus using a technique called “Light Field Photography”. Here, FIG. 18 shows a schematic configuration of a conventional imaging apparatus 100 using this method. The imaging apparatus 100 includes an imaging lens 110, a microlens array 120, and an imaging element 130 in which a plurality of pixels are two-dimensionally arranged. The imaging lens 110 has an aperture stop having a circular opening 101A. 101 is provided. In addition, a plurality of pixels are assigned to one microlens, and imaging data obtained from the imaging element includes information on the light traveling direction in addition to the light intensity distribution on the light receiving surface. . As a result, the image processing unit can reconstruct an observation image from an arbitrary viewpoint and direction (hereinafter simply referred to as a visual field) and an arbitrary focus.

  In such an imaging apparatus 100, since all the light beams passing through the aperture 101A of the aperture stop 101 are received on the imaging element 130 via the microlens array 120, an image (image) received on the imaging element 130 is received. Is similar to the shape of the opening 101A. Therefore, for example, as shown in FIG. 19A, imaging data in which a plurality of circular light reception images are formed in a matrix on the imaging element 130 is obtained. Note that the region S in FIG. 19A is enlarged and shown in FIG.

International Publication No. 06/039486 Pamphlet Ren.Ng and 7 others, “Light Field Photography with a Hand-Held Plenoptic Camera”, Stanford Tech Report CTSR 2005-02

  However, for example, as shown in FIG. 19B, when an aperture stop 101 having a circular opening 101A is used, light is received between the circular light reception images 130D-1 on the image sensor 130. There is a problem that a pixel region (non-light-receiving region 130D-2) that is not formed is generated. For this reason, the imaging pixels cannot be fully utilized.

  The present invention has been made in view of such problems, and an object thereof is to provide an imaging apparatus capable of increasing a light receiving area of an imaging pixel.

  An imaging device according to the present invention is arranged on a focal plane of an imaging lens unit between an imaging lens unit having an aperture stop, an imaging device that acquires imaging data based on received light, and the imaging lens unit. And a cylindrical lens array unit having one cylindrical lens for a plurality of pixels of the image sensor. The aperture stop has a rectangular opening.

  In the imaging apparatus of the present invention, the aperture stop in the imaging lens unit has a rectangular opening, so that an image received on the imaging element via the cylindrical lens array unit is similar to the shape of the opening, It becomes a rectangular shape. Therefore, compared with the case where a conventional aperture stop having a circular opening is used, a plurality of images are easily formed densely in the imaging device without a gap.

  According to the imaging apparatus of the present invention, since the aperture stop in the imaging lens unit has a rectangular opening, the image received on the imaging element is rectangular, and thus has a conventional circular opening. As compared with the case where the aperture stop is used, a plurality of images are easily formed densely in the imaging device without gaps. Therefore, the light receiving area of the imaging pixel can be increased.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[First Embodiment]
FIG. 1 shows the overall configuration of an imaging apparatus (imaging apparatus 1) according to a first embodiment of the present invention. The imaging apparatus 1 captures an imaging target (subject) 2 and outputs imaging data Dout. From the subject 2 side along the positive direction of the Y axis, an aperture stop 10, an imaging lens 11, and the like. A cylindrical lens array 12 and an image sensor 13 are provided. In addition, the imaging apparatus 1 further includes an image processing unit 14, an imaging element driving unit 15, and a control unit 16.

  The aperture stop 10 is an optical aperture stop of the imaging lens 11. The detailed configuration of the aperture stop 10 will be described later (FIG. 2).

  The image pickup lens 11 is a main lens for picking up an image of the subject 2 and is constituted by a general image pickup lens used in, for example, a video camera or a still camera.

  The cylindrical lens array (lenticular lens) 12 is a one-dimensional array (specifically, arrayed along the X-axis direction) of a plurality of cylindrical lenses, which will be described later. f1 represents the focal length of the imaging lens 11). The specific configuration of the cylindrical lens array 12 will be described later (FIG. 3).

  The imaging element 13 receives light from the cylindrical lens array 12 and acquires imaging data D0. The focal plane of the cylindrical lens array 12 (reference numeral f2 in the figure represents the focal length of the cylindrical lens array 12). Are arranged). The imaging device 13 is a two-dimensional array of a plurality of pixels (pixels P described later) arranged in a matrix, and each pixel is configured by, for example, a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), or the like. ing.

  M × N (M, N: integer) pixels are arranged in a matrix on the light receiving surface (surface on the cylindrical lens array 12 side) of such an image pickup device 13, and the cylindrical lens array is arranged for a plurality of pixels. One cylindrical lens within 12 is assigned. The number of pixels on the light receiving surface is, for example, M × N = 3720 × 2520 = 9374400. Here, the number of pixels (m × n) assigned to each cylindrical lens is a resolution in an arbitrary field of view of a reconstructed image described later. For this reason, as the values of m and n increase, the resolution of the reconstructed image at an arbitrary field of view and an arbitrary focus increases. On the other hand, (M / m) and (N / n) are the number of pixels (resolution) of the reconstructed image. For this reason, the number of pixels of the reconstructed image increases as the values of (M / m) and (N / n) increase. Therefore, the resolution and the number of pixels in an arbitrary field of view of the reconstructed image have a trade-off relationship.

  The image processing unit 14 generates and outputs imaging data Dout by performing predetermined image processing on the imaging data D0 obtained by the imaging device 13 in accordance with a control signal Sout supplied from the control unit 16 described later. Is. Specifically, for example, an observation image (reconstructed image) set to an arbitrary field of view or an arbitrary focus is generated by performing arithmetic processing (predetermined rearrangement processing) using a technique called “Light Field Photography”. It is supposed to be.

  The image sensor driving unit 15 drives the image sensor 13 and controls its light receiving operation.

  The control unit 16 controls operations of the image processing unit 14 and the image sensor driving unit 15, and is configured by a microcomputer, for example.

  Next, a specific configuration of the aperture stop 10 will be described with reference to FIG. FIG. 2 is a plan view showing a schematic configuration of the aperture stop 10.

  The aperture stop 10 has a rectangular shape in which the shape of the opening 10A is a point-symmetric figure with respect to the center point of the opening 10A (the optical axis L0 along the Y-axis direction). In the opening 10A, the longitudinal direction (long axis direction) is the X axis direction, and the short side direction (short axis direction) is the Y axis direction.

  Although details will be described later, the F number along the longitudinal direction (X-axis direction) of the opening 10A in the imaging lens 11 is substantially equal to the F number in the arrangement direction (X-axis direction) of the cylindrical lens array 12. It is configured as follows. In the present invention, the F numbers are not limited to being completely the same, and may include errors and the like.

  Next, a specific configuration of the cylindrical lens array 12 will be described with reference to FIG. FIG. 3 is a perspective view showing a schematic configuration of the cylindrical lens array 12.

  The cylindrical lens array 12 is a one-dimensional array of a plurality of cylindrical lenses 12-1 along the longitudinal direction (X-axis direction) of the opening 10 </ b> A in the aperture stop 10. Each cylindrical lens 12-1 has a semi-cylindrical shape (cylinder shape) that extends along the Y-axis direction and protrudes toward the imaging lens 11 side. Each cylindrical lens 12-1 may be configured by a normal lens, or may be configured by a liquid crystal lens, a liquid lens, a diffraction lens, or the like.

  Next, operations and effects of the imaging apparatus 1 according to the present embodiment will be described with reference to FIGS. Here, FIG. 4 is a perspective view for explaining the positional relationship and operation of the aperture stop 10, the cylindrical lens array 12, and the image sensor 13, and FIG. 5 explains the light ray information received by the image sensor 13. FIG. FIG. 6 is a plan view showing a light receiving area in the image sensor 13, and FIG. 7 is a diagram for explaining setting of an F number.

  First, the basic operation of the imaging apparatus 1 will be described with reference to FIGS.

  In the image pickup apparatus 1, as shown in FIG. 4, the image of the subject 2 by the image pickup lens 11 is focused on the cylindrical lens array 12 by the light beam being reduced by the aperture stop 10. An incident light beam to the cylindrical lens array 12 is received by the image sensor 13 through the cylindrical lens array 12. At this time, as shown in FIG. 4, the incident light beam to each cylindrical lens 12-1 in the cylindrical lens array 12 is received at different positions (different pixels P) of the image sensor 13 according to the incident direction. The

Here, with reference to FIG. 5, the light rays received by the image sensor 13 will be described. Considering the one-dimensional coordinate system (u) on the imaging lens surface of the imaging lens 11 and the orthogonal coordinate system (x, y) on the imaging surface of the imaging element 13, the imaging lens surface of the imaging lens 11 and the imaging element 13 are considered. If the distance from the imaging surface is F, the light ray L1 passing through the imaging lens 11 and the imaging device 13 is represented by a three-dimensional function L _F (x, y, u). As a result, the light beam L1 is recorded on the image sensor 13 in a state where the traveling direction of the light beam is maintained in addition to the position information of the light beam. That is, the incident direction of the light beam is determined by the arrangement of the plurality of pixels P assigned to each cylindrical lens 12-1.

  When the image sensor 13 receives light in this way, the image data D0 is obtained from the image sensor 13 in accordance with the drive operation by the image sensor drive unit 15, and the image data D0 is input to the image processing unit 14. In the image processing unit 14, predetermined image processing (rearrangement processing) is performed on the imaging data D <b> 0 under the control of the control unit 16. For example, when an image with an arbitrary field of view is reconstructed, pixel data in the same column is extracted for each cylindrical lens 12-1, and a process of combining them is performed. Further, when reconstructing an image at an arbitrary focus (focus), pixel data rearrangement processing and integration processing are performed. By such image processing, a reconstructed image at an arbitrary field of view or an arbitrary focus is output as the imaging data Dout.

  Next, with reference to FIG. 6 and FIG. 7, the characteristic operation in the imaging apparatus 1 of the present embodiment will be described.

  In this imaging apparatus 1, the shape of the opening 10A of the aperture stop 10 is a rectangle as shown in FIG. 2, and thus the image travels in the plane along the longitudinal direction (X-axis direction) of the opening 10A. The light beam that travels in the plane along the short direction (Y-axis direction) of the opening 10A is collected with a small effective diameter. Thereby, on the image sensor 13, an image (light reception image) in which the effective diameter is changed in accordance with the rotation angle with respect to the optical axis L0 is acquired.

  Therefore, for example, as shown in FIG. 6, a light receiving image (reconstructed pixel region) 13D-1 formed by a light ray incident on the image pickup device 13 via the cylindrical lens array 12 is similar to the shape of the opening 10A. That is, it becomes a rectangular shape whose longitudinal direction is the X-axis direction (long-axis direction). At this time, the light beam in the X-axis direction is decomposed for each incident direction through the cylindrical lens array 12, and the light beam collected with a large effective diameter in the longitudinal direction (X-axis direction) of the opening 10A is also assigned to the number of pixels. It will be divided into fractions. However, by adjusting the ratio of the length in the longitudinal direction (X-axis direction) and the short direction (Y-axis direction) in the opening 10A, it is obtained by condensing the same effective diameter in the X-axis direction and the Y-axis direction. It is possible to obtain a captured image.

  Further, in the cylindrical lens array 12, the plurality of cylindrical lenses 12-1 are arranged in a one-dimensional array (array along the X-axis direction), so that the received light image 13 </ b> D- 1 is placed on the image sensor 13. It is formed in a matrix. As a result, the rectangular light receiving image 13D-1 is arranged in a matrix on the image pickup device 13, and almost the entire area of the image pickup device 13 becomes the light receiving region 13D.

  Here, in the image pickup apparatus 100 using the conventional circular aperture stop (hereinafter referred to as a circular stop) shown in FIGS. 18 and 19, on the image pickup device 130, between the circular light receiving images 130D-1. Since the non-light-receiving region 130D-2 occurs, the imaging pixel (pixel P) cannot be fully utilized. On the other hand, in the imaging device 1 according to the present embodiment, as described above, since almost the entire area of the imaging element 13 becomes the light receiving region 130, the light receiving image is formed densely with no gap compared to the conventional imaging device 100. It becomes easier and the light receiving area increases.

  In the imaging device 1 of the present embodiment, for example, as illustrated in FIG. 6, when a 4 × 1 pixel reconstructed pixel region 13 </ b> D- 1 is set, the image processing unit 14 regenerates an arbitrary viewpoint image. At the time of construction, pixels at the same position in each reconstructed pixel area 13D-1 are extracted, for example, like the extraction pixel P10 shown in FIG.

  As described above, in the present embodiment, since the aperture stop 10 of the imaging lens 11 has the rectangular opening 10A, the image (light reception image 13D-1) received on the imaging element 13 is rectangular. Therefore, compared to the case where a conventional aperture stop having a circular opening is used, the light receiving images 13D-1 are easily formed densely in the image pickup device 13 without a gap. Therefore, the light receiving area of the imaging pixel can be increased.

  Further, in the cylindrical lens array 12, the plurality of cylindrical lenses 12-1 are arranged one-dimensionally along the longitudinal direction (X-axis direction) of the opening 10A in the aperture stop 10, so that the observer can move in the horizontal direction ( When the reconstructed image is viewed along the (X-axis direction), a more suitable reconstructed image can be obtained when observing with the right eye and the left eye.

  Further, when the number of cylindrical lenses 12-1 in the cylindrical lens array 12 is constant, the pixel area (number of pixels P) allocated to each cylindrical lens 12-1 is increased. It is possible to improve the resolution of an arbitrary field of view and an arbitrary focal point in the constructed image. Alternatively, when the number of pixels allocated to each cylindrical lens 12-1 is constant, the number of cylindrical lenses 12-1 increases, so the number of two-dimensional pixels (resolution) of the reconstructed image is increased. Can be made. Therefore, as described above, both the resolution at an arbitrary field of view and an arbitrary focus of the reconstructed image and the number of two-dimensional pixels of the reconstructed image, which are in a trade-off relationship, can be made compatible with the highest possible value. It becomes possible.

In addition, the F number F _ML along the longitudinal direction (X axis direction) of the opening 10A in the imaging lens 11 and the F number F _{SLA in} the arrangement direction (X axis direction) of the cylindrical lens array 12 are substantially equal. Therefore, the phenomenon as shown in FIG. 8 can be avoided. Specifically, for example, as shown in FIG. 8A, when the F number F _ML of the imaging lens 11 is smaller than the F number F _SLA of the cylindrical lens array 12 (when F _ML <F _SLA ). In this case, an overlap occurs between the light rays picked up by the adjacent cylindrical lenses 12-1, thereby causing crosstalk, and the image quality of the reconstructed image is deteriorated. On the other hand, as shown in FIG. 8B, when the F number F _ML of the imaging lens 11 is larger than the F number F _SLA of the cylindrical lens array 12 (F _ML > F _SLA ), the cylindrical Since an imaging pixel in which the imaging light beam by the lens 12-1 is not received is generated, the imaging pixel cannot be sufficiently used, and the number of pixels of the reconstructed image is reduced.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. In addition, the same code | symbol is attached | subjected to the same thing as the component in 1st Embodiment, and description is abbreviate | omitted suitably.

  FIG. 9 illustrates a functional block configuration of the image processing unit (image processing unit 14A) according to the present embodiment. The image processing unit 14A includes a defect correction unit 140, a clamp processing unit 141, a rearrangement processing unit 142, a noise reduction processing unit 143, a white balance processing unit 144, an edge enhancement processing unit 145, and a phase difference detection. A unit 146, an interpolation image synthesis unit 147, a reconstructed image synthesis unit 148, and a gamma correction processing unit 149.

  The defect correction unit 140 corrects a defect such as blackout (a defect caused by an abnormality in the element of the image pickup element 13) included in the image pickup data D0 obtained from the image pickup element 13.

  The clamp processing unit 141 performs black level setting processing (clamp processing) of each pixel data in the image data after defect correction by the defect correction unit 140.

  The rearrangement processing unit 142 generates, for example, each arbitrary viewpoint image by performing predetermined rearrangement processing on the imaging data supplied from the clamp processing unit 142.

  The noise reduction processing unit 143 performs a process of reducing noise included in the imaging data supplied from the rearrangement processing unit 142 (for example, noise generated when imaging is performed in a dark place or a place where sensitivity is insufficient). is there.

  The white balance processing unit 144 performs color balance on the imaging data supplied from the noise reduction processing unit 143 due to the influence of individual device differences such as the pass characteristics of the color filter and the spectral sensitivity of the imaging device 13 and the illumination conditions. Adjustment processing (white balance adjustment processing) is performed. Note that color interpolation processing such as demosaic processing may be performed on the imaging data that has been subjected to white balance processing.

  The contour emphasis processing unit 145 performs contour emphasis processing for emphasizing the image contour on the imaging data D1 supplied from the white balance processing unit 144.

  The phase difference detection unit 146 generates a plurality of parallax images having different parallax (arbitrary viewpoint images from different viewpoints) based on the imaging data supplied from the contour enhancement processing unit 145, and among these parallax images A phase difference DM (for example, a disparity map described later) between at least two parallax images is detected. Such a parallax image is obtained by extracting pixel data (extracted pixels) obtained by pixels P arranged at the same position between unit images (images in the reconstructed pixel region 13D-1) received on the image sensor 13. (P10 pixel data) can be extracted and synthesized. For this reason, the number of generated parallax images is the same as the number of pixels allocated to one cylindrical lens 12-1. Details of the operation of the phase difference detection unit 146 will be described later (FIGS. 10 and 11).

  The interpolated image synthesis unit 147 generates the interpolated synthesized image data D2 using the phase difference DM detected by the phase difference detection unit 146. Specifically, the interpolation image is generated using the phase difference DM, and the parallax image (imaging data D1) supplied from the white balance processing unit 144 and the generated interpolation image are combined to generate the interpolation image. A composite image (interpolated composite image data D2) is generated. The details of the operation of the interpolated image composition unit 147 will be described later (FIGS. 13 and 14).

  The reconstructed image synthesizing unit 148 performs a predetermined synthesizing process on the synthesized image (interpolated synthesized image data D2) supplied from the interpolated image synthesizing unit 147, for example, a refocus calculation process using a technique called “Light Field Photography” To generate a reconstructed image (imaging data D3).

  The gamma correction processing unit 149 generates imaging data Dout by performing predetermined gamma correction (brightness and contrast correction) on the imaging data D3 supplied from the reconstructed image synthesis unit 148.

  Next, with reference to FIGS. 9 to 14, the operation of the imaging apparatus according to the present embodiment (mainly the operation of the image processing unit 14 </ b> A) will be described in detail in comparison with a comparative example. Here, FIG. 10 is a schematic diagram for explaining a phase difference detection operation based on two parallax images, and FIG. 11 is a schematic diagram for explaining pixel correlation calculation using two parallax images. FIG. 12 is a plan view showing a light receiving region in the conventional imaging apparatus 100 according to the comparative example shown in FIGS. 18 and 19. FIG. 13 is a plan view for explaining imaging data (imaging data D0, D1) before image interpolation synthesis according to the present embodiment, and FIG. 14 is an imaging after image interpolation synthesis according to the present embodiment. It is a top view for demonstrating data (imaging data D2).

  In the imaging apparatus of the present embodiment, when the imaging data D0 is input to the image processing unit 14A, defect correction is performed by the defect correction unit 140, and then clamping processing is performed by the clamp processing unit 142. Then, rearrangement processing is performed by the rearrangement processing unit 142, noise reduction processing is performed by the noise reduction processing unit 143, and white balance processing is performed by the white balance processing unit 144, so that the captured image data D1 is contour-enhanced. The data are input to the processing unit 145 and the interpolation image synthesis unit 147, respectively.

  The phase difference detection unit 146 detects a phase difference DM between at least two arbitrary viewpoint images different from each other. Specifically, for example, as shown in FIG. 10, the phase difference Δφ between the viewpoint image by the light beam LR by two parallaxes and the viewpoint image by the light beam LL (the phase φR of the viewpoint image by the light beam LR and the light beam LL). The phase difference from the phase φL of the viewpoint image) is detected.

  More specifically, the phase difference between the viewpoint image VR by the light beam LR and the viewpoint image VL by the light beam LL is calculated as, for example, the following Disparity. For example, as shown in FIG. 11A, a partial image A1 (center coordinate: (x1, y1)) of a small area in the viewpoint image VR is taken out, and a partial image as shown in FIG. A partial image B1 (center coordinates: (x1, y1)) of the same small area as the image A1 is taken out from the viewpoint image VL, and the position of the partial image B1 is moved, and the pixel correlation value by the following equation (1) is Then, the center point of the partial image B1 at the position where the pixel correlation value is maximized is detected as a point corresponding to the center point of the partial image A1. Further, such a calculation process is performed on the entire viewpoint image VR while changing the extraction position of the partial image A1 of the small region, whereby a Disparity Map (set of Disparity) representing the phase difference DM is obtained. can get.

  Next, the interpolated image composition unit 147 generates the interpolated composite image data D2 using the phase difference DM detected by the phase difference detection unit 146. Specifically, an interpolated image is generated using the phase difference DM, and the parallax image (imaging data D1) supplied from the white balance processing unit 144 and the generated interpolated image are combined to generate them. Are synthesized (interpolated synthesized image data D2).

  Next, the reconstructed image composition unit 148 uses a predetermined rearrangement process for the composite image (interpolation composite image data D2) supplied from the interpolation image composition unit 147, for example, a technique called “Light Field Photography”. By performing the refocus calculation process, a reconstructed image (imaging data D3) is generated.

  The gamma correction processing unit 149 performs gamma correction on the reconstructed image data (imaging data D3), thereby generating imaging data Dout and outputting it from the image processing unit 14A.

  Here, in the conventional imaging device 100 according to the comparative example, by using the aperture stop 101 having the conventional circular aperture 101A, for example, by increasing the number of microlenses as shown in FIG. Consider a case where the number of pixels assigned to a lens is set to 2 × 2 = 4. In this case, in each pixel P, the light receiving area has a fan shape. At this time, the traveling direction of the received light beam is different between the region near the arc and the region near the apex angle. Thus, in the light receiving region 130-1 of one pixel P, dispersion in the traveling direction of the received light beam is increased, and it becomes difficult to obtain light beam information in a desired traveling direction. Therefore, by simply reducing the number of pixels assigned to each microlens, when the image is reconstructed by extracting and combining the pixels P, the number of pixels of the reconstructed image can be increased, but the image quality itself is degraded. It will end up.

  Therefore, it is conceivable to generate and interpolate ray space information (perform image interpolation). For this purpose, it is necessary to detect parallax information (phase difference) from ray space information as in the imaging apparatus of the present embodiment. There is. However, in the conventional imaging device 100, as shown in FIG. 12, for example, when the number of microlenses is increased, information on the incident direction is missing in the pixels P at the four corners of the reconstructed pixel region 131D-1. Therefore, it is difficult to perform appropriate image interpolation.

  On the other hand, in the imaging apparatus of the present embodiment, unlike the conventional imaging apparatus 100, the aperture stop 10 of the imaging lens 11 is a rectangular opening 10A as in the first embodiment. The almost entire area of the image sensor 13 becomes the light receiving region 130. Therefore, also in the pixels P at the four corners of the construction pixel region, the lack of information in the incident direction is avoided, and necessary information is reliably acquired. Accordingly, the image processing unit 14A can execute appropriate image interpolation as described above.

  Specifically, for the phase difference detection unit 146 and the interpolation image synthesis unit 147, for example, the imaging data D1 (2 × 1 = 2 having a reconstructed pixel region 13D-2 including two pixels P as shown in FIG. The phase difference DM as described above is detected by the phase difference detecting unit 146 and the interpolated image synthesizing unit 147 generates the interpolated synthesized image data D2 using the phase difference DM. Is done.

  Thus, for example, as in the interpolated composite image data D2 shown in FIG. 14, the viewpoint image (imaging data D1) and the interpolated image (by the interpolated pixel P11) are combined, and the composite image (interpolated composite image data D2). Is generated. Therefore, in order to obtain information on the minimum necessary incident direction, it is sufficient to allocate the minimum number of pixels P to each cylindrical lens 12-1, and the resolution of the reconstructed image is improved.

  As described above, in the present embodiment, even when the number of pixels P assigned to each cylindrical lens 12-1 is reduced to the minimum necessary, ray space information is generated by interpolation using the phase difference DM. Is possible. Therefore, the resolution (spatial resolution) of the reconstructed image can be improved, and at the same time, the number of viewpoints (angular resolution) can be increased by interpolation, thereby improving refocus resolution and wide-viewing stereoscopic vision. It can be realized.

  The imaging apparatus of the present invention includes, for example, a digital camera 3 (Application Example 1) and a stereoscopic display apparatus 4 (Application Example 2) as described below, as well as a camcorder, a position sensor, a biological sensor, an optical microscope, and an FTV (Free viewpoint). It can be applied to an imaging device for uses such as TV).

(Application example 1)
FIGS. 15A and 15B illustrate a schematic configuration of the digital camera 3 on which the imaging device 1 is mounted. FIG. 15A is a front view and FIG. 15B is a side view. The digital camera 3 includes an imaging device 1 (with an aperture stop 10 having a rectangular opening 10A) inside a housing 300. A shutter 17 and a flash are disposed above the housing 300. 18 and a finder optical system 19 are provided.

(Application example 2)
FIG. 16 illustrates a block configuration of the stereoscopic display device 4 on which the imaging device 1 is mounted. The stereoscopic display device 4 displays a stereoscopic image (3D video) of the subject 2, the imaging device 1, a display panel 41 that performs image display based on imaging data Dout output from the imaging device 1, A cylindrical lens array 42 (a plurality of cylindrical lenses arranged one-dimensionally along the X-axis direction) is provided in front of the display panel 41 (between the display panel 4 and the viewer 5). . In addition, as the display panel 41, a liquid crystal panel, an organic EL (Electro Luminescence) panel, etc. are used, for example.

  In this stereoscopic display device 4, by recording light rays emitted from each point of the subject 2 in all directions, the same light field as the light rays emitted from the subject 2 is reproduced at the time of display. As a result, the viewer 5 can view the same light beam as when the subject 2 is actually viewed, and thus can be viewed as a stereoscopic image. In this stereoscopic display device 4, the lens array (cylindrical lens arrays 12, 42) is on the opposite side during shooting and during display, and thus processing for reversing the left and right positions is required for each element image. .

  While the present invention has been described with reference to the first and second embodiments and application examples, the present invention is not limited to these embodiments and the like, and various modifications are possible.

  For example, in the above-described embodiment and the like, the image processing unit 14 has been described as one of the components of the imaging device 1, but the image processing unit is not necessarily provided inside the imaging device. Specifically, the image processing unit is provided in a device different from the imaging device, such as a PC (Personal Computer), and the imaging data obtained by the imaging device is transferred to the PC. It is also possible to perform processing.

  In the above-described embodiment and the like, the phase difference detection unit 146 detects the phase difference between the two based on two parallax images having different parallaxes. However, the present invention is not limited to this, and three or more parallaxes are detected. The phase difference may be detected based on the image.

  Moreover, in the said embodiment etc., it is set as the structure which has arrange | positioned the position of the aperture stop 10 in the imaging target object 2 side (incident side) of the imaging lens 11, However, it is not limited to this, The image side ( (Emission side) Alternatively, a configuration provided inside the imaging lens 11 may be used.

  In the above-described embodiment and the like, in the opening 10A of the aperture stop 10, the longitudinal direction (major axis direction) is the X axis direction and the short direction (minor axis direction) is the Y axis direction. The shape of the opening is not limited to this. That is, for example, like the opening 10A-1 in the aperture stop 10-1 shown in FIG. 17, the longitudinal direction (major axis direction) is the Y axis direction, and the short direction (minor axis direction) is the X axis direction. It may be. In such a configuration, the light emitted from one point of the subject 2 passes through the entire wide aperture area of the imaging lens 11 and is condensed on the cylindrical lens array 12. At this time, the cylindrical lens 12 causes the light to converge in the X-axis direction. However, since the light beams in the Y-axis direction are integrated while being overlapped and acquired by the image sensor 13, brighter imaging data can be obtained.

  In the above-described embodiment and the like, the case where the plurality of cylindrical lenses 12-1 are arranged one-dimensionally along the X-axis direction in the cylindrical lens array 12 has been described. However, the shape and arrangement of the cylindrical lenses include Not limited. Specifically, for example, a plurality of cylindrical lenses may be arranged one-dimensionally along the Y-axis direction.

  Further, on the light receiving surface of the image sensor 13, for example, a color filter (not shown) may be color-coded for each pixel P and arranged two-dimensionally. As such a color filter, for example, color filters of three primary colors of red (R), green (G), and blue (B) are arranged in a checkered pattern at a ratio of R: G: B = 1: 2: 1. Alternatively, a Bayer color filter (primary color filter) can be used. If such a color filter is provided, the image data D0 obtained by the image sensor 13 can be pixel data of a plurality of colors (in this case, three primary colors) corresponding to the color of the color filter, The reconstructed image can be a color image. At this time, unlike the conventional imaging apparatus 100, since almost the entire area of the image sensor 13 becomes the light receiving area 130, there is no non-light receiving area like 130D-2 in FIG. This can be done correctly, and the occurrence of false colors can be suppressed. In addition, if a color filter color-coded for each pixel region corresponding to each cylindrical lens 12-1 is used, when pixels arranged at the same position for each cylindrical lens 12-1 are extracted, even after extraction. The same color arrangement can be used. Therefore, for example, processing such as color interpolation can be easily performed, and generation of false colors can be suppressed.

It is a figure showing the whole imaging device composition concerning a 1st embodiment of the present invention. It is a top view showing schematic structure of the aperture stop shown in FIG. It is a perspective view showing schematic structure of the cylindrical lens array shown in FIG. It is a perspective view for demonstrating the positional relationship and effect | action in an aperture stop, a cylindrical lens array, and an image pick-up element. It is a figure for demonstrating the information of the light ray which injects into an image sensor. It is a top view showing the light reception area | region on an image pick-up element at the time of using the aperture stop shown in FIG. It is a top view for demonstrating the extraction pixel at the time of reconstructing an arbitrary viewpoint images. It is a figure for demonstrating the setting of F number of an imaging lens and a cylindrical lens array. It is a functional block diagram showing the structure of the image process part which concerns on the 2nd Embodiment of this invention. It is a schematic diagram for demonstrating the phase difference detection operation | movement based on two parallax images. It is a schematic diagram for demonstrating pixel correlation calculation by two parallax images. It is a top view showing the light reception area | region in the conventional imaging device which concerns on a comparative example. It is a top view for demonstrating the imaging data before the image interpolation composition which concerns on 2nd Embodiment. It is a top view for demonstrating the imaging data after the image interpolation composition which concerns on 2nd Embodiment. It is a figure showing the external appearance structure of the digital camera which is one application example of the imaging device of this invention. It is a figure showing the structure of the stereoscopic display apparatus which is the other application example of the imaging device of this invention. It is a perspective view for demonstrating the structure and effect | action of an aperture stop which concern on the modification of this invention. It is a figure showing the whole structure of the conventional imaging device. It is a figure showing an example of the imaging data by the conventional imaging device shown in FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Imaging device 10, 10-1 ... Aperture stop, 10A, 10A-1 ... Aperture, 11 ... Imaging lens, 12 ... Cylindrical lens array, 12-1 ... Cylindrical lens, 13 ... Imaging element, 13D ... Light receiving area , 13D-1 ... received light image (reconstructed pixel region), 13D-2, 13D-3 ... reconstructed pixel region, 14, 14A ... image processing unit, 140 ... defect correction unit, 141 ... clamp processing unit, 142 ... array Replacement processing unit, 143 ... Noise reduction processing unit, 144 ... White balance processing unit, 145 ... Outline enhancement processing unit, 146 ... Phase difference detection unit, 147 ... Interpolated image synthesis unit, 148 ... Reconstructed image synthesis unit, 149 ... Gamma Correction processing unit, 15 ... imaging element driving unit, 16 ... control unit, 17 ... shutter, 18 ... flash, 19 ... finder optical system, 2 ... imaging object (subject), 3 Digital camera, 300 ... casing, 4 ... stereoscopic display device, 41 ... display panel, 42 ... cylindrical lens array, 5 ... viewer, f1, f2 ... focal length, Sout ... control signal, D0, D1-D3, Dout ... Imaging data (image data), DM: phase difference information (Disparity Map), P: pixel, P10: extracted pixel, P11: interpolation pixel, F _ML , F _SLA ... F number (F value), VR, VL ... parallax image (Arbitrary viewpoint image), A1, B1 ... partial image, L0 ... optical axis, L1 ... light ray, LR, LL ... light ray of parallax image, φR, φL ... phase of parallax image, Δφ ... phase difference.

Claims (7)

An imaging lens unit having an aperture stop;
An image sensor that acquires imaging data based on the received light;
A cylindrical lens array unit disposed on a focal plane of the imaging lens unit between the imaging lens unit and the imaging device, and having a cylindrical lens for a plurality of pixels of the imaging device;
An imaging apparatus in which the aperture stop has a rectangular opening. The imaging device according to claim 1, wherein a plurality of cylindrical lenses are arranged along a longitudinal direction of the opening. The imaging apparatus according to claim 1, wherein an F number of the imaging lens unit is equal to an F number of the cylindrical lens array. The imaging apparatus according to any one of claims 1 to 3, further comprising an image processing unit that performs a predetermined rearrangement process based on imaging data obtained from the imaging element and generates a reconstructed image. The image processing unit generates a plurality of parallax images having different parallax from each other based on the imaging data, detects a phase difference between at least two parallax images among the plurality of parallax images, and detects the detected phase difference. The imaging apparatus according to claim 4, wherein the reconstructed image is generated using the imaging apparatus. The image processing unit generates an interpolated image using the phase difference, and generates the reconstructed image by performing the rearrangement process on a composite image of the parallax image and the interpolated image. Item 6. The imaging device according to Item 5. The imaging device according to claim 1, further comprising a color filter that is color-coded for each pixel on a light receiving surface side of the light receiving element.

Images