JP4730347B2 - Imaging apparatus and imaging method - Google Patents

Description

  The present invention relates to an imaging apparatus and an imaging method, and more specifically, for example, two imaging modes of an imaging mode based on a light field photography technique and a normal high-resolution imaging mode. The present invention relates to an image pickup apparatus and an image pickup method capable of picking up images.

  Conventionally, various imaging devices have been proposed and developed. There has also been proposed an imaging apparatus in which predetermined image processing is performed on an imaging signal obtained by imaging and output. For example, Patent Document 1 and Non-Patent Document 1 propose an imaging apparatus using a technique called light field photography. This imaging device is composed of an imaging lens, a microlens array, a light receiving element, and an image processing unit. An imaging signal obtained from the light receiving element is in addition to the light intensity on the light receiving surface of the light receiving element. It also contains information on the direction of travel of the light. Then, based on such an imaging signal, an observation image from an arbitrary viewpoint or direction is reconstructed in the image processing unit.

International Publication No. 06/039486 Pamphlet Ren.Ng, et al, "Light Field Photography with a Hand-Held Plenoptic Camera", Stanford Tech Report CTSR 2005-02

  By the way, in an imaging apparatus using light field photography technology, a normal high resolution imaging mode (referred to as a first imaging mode) that does not use light field photography technology, and light field photography. It is conceivable that the imaging mode based on technology (referred to as a second imaging mode) is used by switching as appropriate. However, in order to switch between the two imaging modes in this way, it is necessary to mechanically move the microlens array. That is, in the first imaging mode, since the microlens array is unnecessary, it is necessary to move the microlens array from the optical axis and remove it from the optical axis. On the other hand, in the second imaging mode, it is necessary to arrange the microlens array on the optical axis. Therefore, when attempting to switch between the two imaging modes, there is a problem that the configuration and structure of the imaging apparatus become complicated and the entire imaging apparatus becomes large.

  Therefore, an object of the present invention is to have a simple configuration and structure, and for example, it is possible to easily switch between two imaging modes, for example, an imaging mode based on light field photography technology and a normal high-resolution imaging mode. An imaging device and an imaging method using the imaging device are provided.

In order to achieve the above object, an imaging apparatus of the present invention provides:
(A) an imaging lens,
(B) a first imaging element disposed on the focal plane of the imaging lens;
(C) a second image sensor, and
(D) imaging means;
With
Imaging is performed in the first imaging mode and the second imaging mode,
During imaging in the first imaging mode, light that has passed through the imaging lens forms an image on the imaging surface of the first imaging element,
During imaging in the second imaging mode, the light that has passed through the imaging lens passes through the first imaging element, and reaches the imaging surface of the second imaging element by the imaging unit.

In order to achieve the above object, the imaging method of the present invention is an imaging apparatus of the present invention, that is,
(A) an imaging lens,
(B) a first imaging element disposed on the focal plane of the imaging lens;
(C) a second image sensor, and
(D) imaging means;
An imaging method using an imaging apparatus comprising:
At the time of imaging in the first imaging mode, the light passing through the imaging lens is imaged on the imaging surface of the first imaging element,
At the time of imaging in the second imaging mode, the light that has passed through the imaging lens is allowed to pass through the first imaging element and reach the imaging surface of the second imaging element by the imaging unit.

In an imaging device used in the imaging device or imaging method of the present invention (hereinafter collectively referred to as “imaging device of the present invention”), the imaging means is provided in the first imaging device. A configuration including a microlens array portion may be employed. Such a configuration is referred to as a first configuration for convenience. Here, the microlens array unit is provided in the first image sensor. Specifically, the microlens array unit may be provided integrally with the first image sensor, or the microlens array unit and After the first image sensor is manufactured separately, the microlens array unit and the first image sensor may be integrated (for example, bonded). In the first configuration, the microlens array unit may face the imaging lens (that is, from the imaging lens side, the microlens array unit and the first imaging element are arranged in this order). However, it is facing the second image sensor (that is, the first image sensor and the microlens array section are arranged in this order from the imaging lens side), and the design freedom Since the degree can be increased, it is more preferable. Even when the microlens array unit faces the imaging lens, the influence exerted by the microlens array unit is ignored when the light passing through the imaging lens is imaged on the imaging surface of the first imaging element. As small as possible. However, depending on the case, the image processing unit to be described later may perform processing for removing the influence exerted by the microlens array unit. Alternatively, in the first configuration, the second imaging element is composed of a plurality of light receiving elements (referred to as second light receiving elements for convenience), and the microlens array unit is composed of a plurality of microlenses. It is preferable that a plurality of second light receiving elements constituting the second imaging element correspond to one microlens. Examples of the number of second light receiving elements constituting the second imaging element corresponding to one microlens include 4 to 4096. The microlens is preferably composed of a plano-convex lens, but is not limited to this, and may be a biconvex lens. When the microlens is configured from a plano-convex lens, the convex portion of the plano-convex lens may face the second imaging element or may face the imaging lens. In the micro lenses constituting the microlens array section, the curvature of the surface of the imaging lens side (r ₁₎ than it is preferably towards the curvature of the second image pickup element side surface (r ₂₎ is greater. Here, the curvature is a value at a point where the optical axis of the microlens and the surface of the microlens intersect. The microlens can be a spherical lens or an aspherical lens as described above, but is not limited to this. For example, a zone plate, a holographic lens, a kinoform lens, or a binary optical element can be used. It can also be a diffractive lens exemplified. Further, 4 to 4096 can be exemplified as the number of light receiving elements (referred to as first light receiving elements for convenience) constituting the first imaging element corresponding to one microlens.

  Alternatively, in the imaging device or the like of the present invention, the imaging means can be configured by a plurality of pinholes provided in the first imaging element. Such a configuration is referred to as a second configuration for convenience. And in this 2nd structure, the 2nd image pick-up element is comprised from the several 2nd light receiving element, and the 2nd light receiving element which comprises a 2nd image pick-up element in one pinhole. It is preferable that a plurality correspond. Examples of the number of second light receiving elements constituting the second image sensor corresponding to one pinhole include 4 to 4096. Examples of the number of first light receiving elements constituting the first imaging element corresponding to one pinhole include 4 to 4096.

  In the imaging device or the like of the present invention including the preferred configuration described above, the first imaging element can be configured to pass only light having a specific wavelength (for example, infrared). In this case, the second image sensor may have a sensitivity to a specific wavelength that has passed through the first image sensor. In such a form, if the first image sensor is an image sensor provided with a color filter and the second image sensor is an image sensor from which the color filter is removed, a reduction in resolution in the second image sensor is reduced. It becomes possible. In order to pass only light having a specific wavelength (for example, infrared light), when an infrared cut filter for cutting infrared light is provided in front of the first image sensor, the light is transmitted through the light transmitting portion of the first image sensor. An infrared cut filter is provided in the vicinity of the first image sensor so as not to cut the infrared light with respect to the light and so as to cover other than the light transmission part of the first image sensor, and the first image sensor An infrared transmission filter that selectively allows only infrared rays to pass therethrough may be provided between the first image sensor and the second image sensor. Further, in order to obtain a configuration having sensitivity to a specific wavelength that has passed through the first imaging device, for example, in the case of infrared rays, a CCD (Charge Coupled Device) is used as the second imaging device. An image sensor having sensitivity to infrared rays such as the above may be used.

Furthermore, in the imaging apparatus of the present invention including the preferred configuration and form described above,
An image processing unit for performing predetermined image processing on the signal from the second image sensor;
During imaging in the first imaging mode, image processing by the image processing unit is stopped,
It can be configured such that image processing by the image processing unit is performed during imaging in the second imaging mode.

  Examples of the imaging lens in the imaging apparatus according to the present invention including various preferable configurations and forms described above include general imaging lenses used in video cameras, still cameras, and the like. The first imaging element (first imaging means) includes, for example, a plurality of TFTs (Thin Film Transistors) arranged in a two-dimensional matrix and a first light receiving element combined with each TFT. It can be configured (see, for example, Japanese Patent Publication No. 2778818 or Japanese Patent Publication No. 28337578). Alternatively, as a combination of the first imaging element and the imaging means, a plurality of CCDs and CMOS sensors (first light receiving elements) arranged in a two-dimensional matrix and pins incorporated in each CCD and CMOS sensor Halls can be listed. Examples of the second imaging element (second imaging means) include a plurality of CCDs and CMOS sensors (second light receiving elements) arranged in a two-dimensional matrix. Alternatively, as the second image sensor, a solid-state image sensor that has a configuration disclosed in Japanese Patent Laid-Open No. 5-48060 and achieves a color filter function with liquid crystal may be used.

  Examples of materials constituting the microlens array section include plastics such as polymethyl methacrylate resin (PMMA), polycarbonate resin (PC), polyarylate resin (PAR), polyethylene terephthalate resin (PET), acrylic resin, and ABS resin. In addition, glass can be mentioned and it can be produced by a known method. The microlens array section is formed by arranging a plurality of microlenses in a two-dimensional matrix. The microlens array portion may be partitioned into a plurality of regions, and the focal length of the microlens may be changed for each partition, or the diameter of the microlens may be changed. A plurality of second light receiving elements correspond to one microlens or pinhole, but the plurality of second light receiving elements are made into a block (segmented) to form a second Imaging data may be generated based on imaging information obtained from the light receiving element. It is desirable that the light emitted from one microlens or pinhole does not enter the second light receiving element corresponding to the microlens or pinhole adjacent to the microlens or pinhole.

  In the imaging device or the like of the present invention, an image by the imaging lens is formed on the first imaging element in the first imaging mode (and when imaging in the second imaging mode as desired). However, such an imaging state can be easily performed by optimizing the focusing operation in the imaging lens. Specifically, for example, in auto-focus processing, it is only necessary to optimize the focusing operation (processing) during imaging in the first imaging mode (and the second imaging mode).

  According to the imaging apparatus or imaging method of the present invention, when imaging in the first imaging mode (when imaging in the normal high-resolution imaging mode), the light that has passed through the imaging lens is on the imaging surface of the first imaging element. When imaging in the second imaging mode (for example, imaging in the imaging mode based on light field photography technology), the light passing through the imaging lens passes through the first imaging element, and Since the imaging unit reaches the imaging surface of the second imaging device, the first imaging mode and the second imaging mode are not moved without moving the first imaging device, the second imaging device, or the imaging unit. The imaging mode can be switched between the imaging modes. Further, since the same imaging optical system is used between these two imaging modes, the configuration and structure of the imaging apparatus are not complicated.

  Hereinafter, the present invention will be described based on examples with reference to the drawings.

  Example 1 relates to an imaging apparatus and an imaging method of the present invention. More specifically, the imaging device of Embodiment 1 has the first configuration. FIG. 1 shows a conceptual diagram of the image pickup apparatus 1 according to the first embodiment.

The imaging device 1 according to the first embodiment captures an imaging target and outputs imaging data _Dout .
(A) Imaging lens 11,
(B) a first imaging device (first imaging means) 12 disposed on the focal plane of the imaging lens;
(C) a second imaging device (second imaging means) 14, and
(D) imaging means;
It has.

  In the imaging apparatus 1 of the first embodiment, imaging is performed in the first imaging mode (normal high-resolution imaging mode) and the second imaging mode (imaging mode based on the light field photography technique). The light that has passed through the imaging lens 11 during imaging in the first imaging mode is imaged on the imaging surface of the first imaging element 12, and has passed through the imaging lens 11 during imaging in the second imaging mode. The light passes through the first image sensor 12 and reaches the imaging surface of the second image sensor 14 by the imaging means.

Here, in the first embodiment, the image forming unit includes the microlens array unit 13 provided in the first image sensor 12. The microlens array unit 13 is integrally provided in the first image sensor 12 so as to face the second image sensor 14. The second imaging element 14 is composed of a plurality of second light receiving elements, and the microlens array unit 13 is composed of a plurality of microlenses 13 ′. A plurality of second light receiving elements constituting the imaging element 14 correspond to each other. Specifically, the number of second light receiving elements constituting the second imaging element 14 corresponding to one microlens is, for example, 4 to 4096. The micro lens 13 ′ is composed of a plano-convex lens. That is, in the microlens 13 ′ constituting the microlens array unit 13, the curvature (r ₂ ) of the surface on the second imaging element side is larger than the curvature (r ₁ ) of the surface on the imaging lens 11 side. Furthermore, the microlens 13 ′ is composed of an aspheric lens. Further, the number of first light receiving elements constituting the first imaging element 12 corresponding to one microlens 13 ′ is, for example, 4 to 4096.

  The imaging apparatus 1 according to the first embodiment further includes an image processing unit 15 for performing predetermined image processing on the signal from the second imaging element 14, and control for controlling the image processing unit 15. A portion 17 is further provided. Then, image processing by the image processing unit 15 is stopped at the time of imaging in the first imaging mode, and image processing by the image processing unit 15 is performed at the time of imaging in the second imaging mode.

In Example 1 or Example 2 to be described later, the imaging lens 11 is a main lens for imaging an imaging object, and includes, for example, a general imaging lens used in a video camera, a still camera, or the like. Has been. The first image sensor 12 is composed of a plurality of TFTs arranged in a two-dimensional matrix and a first light receiving element composed of a MIS diode combined with each TFT. The first image sensor 12 is disposed on the focal plane of the imaging lens 11, although depending on the in-focus state of the imaging lens 11. In the figure, the symbol f ₁ represents the distance from the center of the imaging lens 11 to the imaging plane of the first imaging element 12. During imaging in the first imaging mode, the first imaging element 12 receives light emitted from the imaging lens 11 to generate a first imaging signal, and the first imaging signal is an image processing unit. 15 is sent. The microlens array unit 13 is formed by arranging a plurality of microlenses 13 ′ in a two-dimensional matrix.

The second image sensor 14 is composed of, for example, a plurality of CCDs (second light receiving elements) arranged in a two-dimensional matrix. During imaging in the second imaging mode, the second imaging element 14 receives the light emitted from the microlens array unit 13, generates a second imaging signal, and sends it to the image processing unit 15. Image processing is performed. The second image sensor 14 is disposed on the focal plane of the microlens array unit 13. In the figure, symbol f ₂ represents the distance from the center of the microlens array unit 13 to the imaging surface of the second imaging element 14 (focal length of the microlens 13 ′).

  The first image pickup device 12 is driven by the image pickup device driving means 16 to control the light receiving operation of the first image pickup device 12, and further, the second image pickup device 14 is driven and the second image pickup device is driven. 14 light receiving operations are controlled. The control unit 17 controls the operations of the image processing unit 15 and the image sensor driving unit 16. Specifically, the control unit 17 appropriately controls the driving operation of the image sensor driving unit 16, and the operation of the image processing unit 15 according to the two imaging modes of the first imaging mode and the second imaging mode. To control. The control unit 17 is composed of a microcomputer.

In Example 1 or Example 2 described later, in the second imaging mode, the image processing unit 15 performs image processing. The image processing unit 15 performs predetermined image processing on the signal (second imaging signal) obtained by the second imaging element 14 in the second imaging mode, and outputs the imaged data _Dout . Specifically, a refocusing calculation process based on the light field photography technique is performed. Thus, an observation image from an arbitrary viewpoint or direction can be reconstructed, and three-dimensional information of the image can be acquired. The refocus calculation process will be described later.

FIG. 2 shows a schematic partial cross-sectional view of the microlens array unit 13 in the first embodiment. Here, FIG. 2 is a schematic partial cross-sectional view for explaining the lens effect of the microlens array unit 13. In the microlens 13 ′ constituting the microlens array unit 13, the curvature (r ₂ ) of the surface on the second imaging element 14 side is larger than the curvature (r ₁ ) of the surface on the imaging lens 11 side. Specifically, in Example 1, r ₁ / r ₂ = 0. The microlens 13 ′ constituting the microlens array unit 13 is a plano-convex lens that is convex toward the second image sensor. Reference numeral 13 ″ indicates the base of the microlens array unit 13.

  When natural light including light in various wavelength regions is used for imaging as in the imaging device 1 of the first embodiment, the surface of the microlens 13 ′ on the second imaging element 14 side is aspherical, The lens 13 ′ is preferably an aspheric lens. As a result of the increased curvature as compared with the case of using a spherical lens, optical design is facilitated. Further, compared to the case where the microlens 13 ′ is configured by a diffractive lens, the wavelength dependency when the incident light is refracted is eliminated, so that it is possible to avoid the occurrence of axial chromatic aberration and the like, and light in various wavelength regions. It can be set as the structure suitable for the imaging by natural light containing. In the case of an imaging application using monochromatic light or the like, there is no problem of wavelength dependency and axial chromatic aberration. In some cases, excellent optical characteristics can be obtained.

  With reference to FIGS. 1, 2, and 3A and 3B, the operation of the imaging apparatus 1 according to the first embodiment will be described in detail.

  In the imaging apparatus 1 of the first embodiment, in the first imaging mode (normal high-resolution imaging mode), the light that has passed through the imaging lens 11 is imaged on the imaging surface of the first imaging element 12. Let That is, the image of the object to be imaged by the imaging lens 11 is formed on the first imaging element 12 without being affected by the microlens array unit 13. In this way, the first image signal is obtained from the first image sensor 12 under the control of the image sensor driving means 16.

On the other hand, in the second imaging mode (imaging mode based on light field photography technology), the light that has passed through the imaging lens 11 passes through the first imaging element 12 and is a microlens that is an imaging means. The micro lens 13 ′ constituting the array unit 13 is caused to reach the imaging surface of the second imaging element 14. In this way, a second image signal is obtained from the second image sensor 14 under the control of the image sensor driving means 16. That is, in the second imaging mode, as shown in FIG. 2, the incident light L ₁₁ incident on the micro lens 13 ′ is refracted by the micro lens 13 ′ and is a pixel that is a focal point on the optical axis L _0. Focused on PL ₁₁ . As described above, during imaging in the second imaging mode, the image formed by the imaging lens 11 formed on the microlens 13 ′ can be re-imaged (condensed and converged) on the second imaging element 14. it can.

  Actually, information based on two imaging modes (a normal high-resolution imaging mode and an imaging mode based on the light field photography technology) can be obtained at the same time. Is not necessary. That is, there is no need to actively switch between the two imaging modes, and switching between the two imaging modes actually corresponds to switching of image processing in the image processing unit 15.

As shown in FIG. 2, the incident light L ₁₁ (shown by a solid line) on the microlens array unit 13 forms an image at a point (pixel) PL ₁₁ on the second image sensor 14 and enters the microlens array unit 13. Incident light L ₁₂ (indicated by a dotted line) forms an image at a point (pixel) PL ₁₂ on the second image sensor 14, and incident light L ₁₃ (indicated by a one-dot chain line) on the microlens array unit 13 is An image is formed on a point (pixel) PL ₁₃ on the second image sensor 14. That is, when the incident direction of the incident light on the microlens array unit 13 is different, an image is formed (condensed) on a different point (different pixel) on the second image sensor 14.

  Such an imaging state can be easily performed by optimizing the in-focus operation in the imaging lens 11. For example, in auto-focus processing, imaging in the first imaging mode and imaging in the second imaging mode are possible. What is necessary is to optimize the focusing operation (processing) at the time. That is, for example, the light passing through the imaging lens 11 can be imaged on the imaging surface of the first imaging element 12 regardless of whether the imaging is performed in the first imaging mode or the second imaging mode. That's fine. The same applies to Example 2 described later.

The first image signal obtained by the first image sensor 12 and the second image signal obtained by the second image sensor 14 are sent to the image processing unit 15. In the image processing unit 15, under the control of the control unit 17, predetermined image processing is performed on the imaging signal and output as imaging data _Dout . Specifically, in the first imaging mode, as a result of stopping the image processing by the image processing unit 15 under the control of the control unit 17, the input first imaging signal is output as it is as the imaging data _Dout. The On the other hand, in the second imaging mode, as a result of image processing (refocus calculation processing) performed by the image processing unit 15 under the control of the control unit 17, a predetermined image is input to the input second imaging signal. Processing is performed and output as imaging data _Dout .

  Here, with reference to FIGS. 3A and 3B, the refocusing calculation processing of the image processing in the image processing unit 15 will be described in detail. This refocus calculation process is similarly applied to Example 2 described later.

As shown in FIG. 3A, an orthogonal coordinate system (u, v) is assumed on the imaging lens surface 11A of the imaging lens 11, and an orthogonal coordinate system (x) on the imaging surface 14A of the second imaging element 14 is assumed. , Y). If the distance between the imaging lens surface of the imaging lens 11 and the imaging surface of the second imaging element 14 is f, the light rays passing through the imaging lens 11 and the second imaging element 14 as shown in FIG. L ₁₄ can be represented by a four-dimensional function L _F (x, y, u, v). Therefore, in addition to the position information of the light beam L _14, it is possible to obtain information on the traveling direction of light L _14. In this case, as shown in FIG. 3B, the imaging lens surface 11A, the imaging surface 14A, and the refocusing surface (the imaging surface of the first imaging element 12 on which the image by the imaging lens 11 is formed or the microlens When the positional relationship between the image planes 13A of the array unit 13) is set, that is, when the refocus plane 13A is set so that f ′ = α · f, the coordinates (s, t on the refocus plane 13A are set. The detected light intensity L _{F ′} (s, t, u, v) on the imaging surface 14A can be expressed by the following equation (1). Further, the image E _{F ′} (s, t) obtained on the refocus surface 13A is obtained by integrating the detected light intensity L _{F ′} (s, t, u, v) with respect to the lens aperture, and therefore, 2). Therefore, by performing the refocusing operation based on the equation (2), an observation image from an arbitrary viewpoint or direction can be reconstructed by using the imaging data _Dout based on the light field photography technique. The three-dimensional information can be acquired.

  As described above, in the first embodiment, during imaging in the first imaging mode (when imaging in the normal high-resolution imaging mode), the light that has passed through the imaging lens 11 is captured by the first imaging element 12. When imaged on the surface and imaged in the second imaging mode (imaged in the imaging mode based on the light field photography technology), the light passing through the imaging lens 11 passes through the first imaging element 12. In addition, the imaging means 13 reaches the imaging surface of the second imaging element 14. Therefore, it is possible to easily switch the imaging mode between the first imaging mode and the second imaging mode with a simple configuration and structure. In addition, since information based on two imaging modes can be obtained at the same time, and it is not necessary to substantially switch the imaging mode, the reliability during the imaging mode switching operation is improved compared to the case where the imaging mode is mechanically switched. Can be improved. Furthermore, if the microlens 13 ′ is composed of an aspheric lens, the curvature can be increased as compared with the case where the microlens 13 ′ is composed of a spherical lens, so that optical design can be facilitated.

  In some cases, as shown in a conceptual diagram of the imaging apparatus in FIG. 4, the microlens array unit 13 may be provided in the first imaging element 12 so as to face the imaging lens 11.

  The second embodiment is a modification of the first embodiment and relates to the second configuration. FIG. 5 shows a conceptual diagram, and a schematic partial cross-sectional view in which a part of the first image sensor and the imaging means is enlarged is shown in FIG. The image unit includes a plurality of pinholes 113 provided in the first image sensor (first image sensor) 112. Even in the second embodiment, the second imaging element (second imaging means) 14 is composed of a plurality of second light receiving elements, and the second imaging element 14 is placed in one pinhole 113. A plurality of second light receiving elements to be configured correspond to each other. Specifically, 4 to 4096 can be exemplified as the number of second light receiving elements constituting the second imaging element 14 corresponding to one pinhole 113. In the second embodiment, the first image sensor 112 is composed of a plurality of CCDs arranged in a two-dimensional matrix. More specifically, the CCD in the first image sensor 112 includes a light receiving unit 120, a transfer path 121, and a light shielding unit 122, as shown in a schematic partial cross-sectional view in FIG. A pinhole 113 is provided in an area where the CCD is not provided. Further, the number of first light receiving elements constituting the first image sensor 12 corresponding to one pinhole 113 is, for example, 4 to 4096.

  Even in the imaging apparatus 2 according to the second embodiment, during imaging in the first imaging mode (when imaging in the normal high-resolution imaging mode), light that has passed through the imaging lens 11 is captured by the first imaging element 112. The image is formed on the surface. On the other hand, during imaging in the second imaging mode (when imaging in the imaging mode based on the light field photography technology), the light that has passed through the imaging lens 11 passes through the first imaging element 112 and forms an image. It reaches the imaging surface of the second imaging element 14 by means of a pinhole 113 as means.

  Except for the above points, the configuration and structure of the imaging apparatus 2 according to the second embodiment can be the same as the configuration and structure of the imaging apparatus 1 according to the first embodiment, and thus detailed description thereof is omitted.

  As mentioned above, although this invention was demonstrated based on the preferable Example, this invention is not limited to these Examples, A various deformation | transformation is possible. In the embodiment, the refocus calculation process based on the light field photography technique has been described as the image processing method in the image processing unit 15, but the image processing method in the image processing unit 15 is not limited to this, Other image processing methods (for example, image processing such as shaking the field of view, or image processing such as distance calculation obtained by causing the microlens array unit and the image sensor to function as a kind of stereo camera) may be used.

  The microlens array unit in the first embodiment may be configured from a liquid crystal lens array, or may be configured from a liquid microlens array unit using an electrowetting phenomenon (electrocapillary phenomenon). The first imaging elements 12 and 112 are not limited to light receiving elements and CCDs combined with TFTs, and have a function of transmitting or passing light and converting received light into electrical signals. Any type of image sensor can be used as long as it is one. For example, if the liquid crystal lens is configured based on the technology disclosed in Japanese Patent No. 2788818 or Japanese Patent No. 28337578, the microlens array unit and the first imaging element can be integrated.

  The first imaging elements 12 and 112 may be configured to pass only light having a specific wavelength (for example, infrared). In this case, the second image sensor 14 may be configured to be sensitive to a specific wavelength that has passed through the first image sensors 12 and 112. Specifically, the first image sensors 12 and 112 are arranged close to the first image sensor so that infrared rays are not cut for light transmitted through the light transmission portion of the first image sensor, and Light transmitted through the first image sensor provided between the first image sensor and the CCD in which the infrared cut filter is arranged so as to cover other than the light transmission part of the first image sensor. Of these, an infrared transmission filter that selectively transmits only infrared light may be used, and the second image pickup device 14 may be formed of an image pickup device such as a CCD having sensitivity to infrared light. Thus, by separating the wavelengths used by the two image sensors, the second image sensor 14 can receive light efficiently. Furthermore, if the second image sensor is a monochrome image sensor from which the color filter is removed, high resolution for the pixels allocated for color detection can be realized.

FIG. 1 is a conceptual diagram of the imaging apparatus according to the first embodiment. FIG. 2 is a schematic partial cross-sectional view enlarging the microlens array part for explaining the lens effect of the microlens array part. 3A is a conceptual diagram of an imaging lens and the like for explaining image processing in the second imaging mode, and FIG. 3B is a diagram for explaining image processing in the second imaging mode. It is a figure for doing. FIG. 4 is a conceptual diagram of a modification of the imaging apparatus according to the first embodiment. FIG. 5 is a conceptual diagram of the imaging apparatus according to the second embodiment. FIG. 6 is a schematic partial cross-sectional view in which the first image sensor and the pinhole are enlarged.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1, 2 ... Imaging device, 11 ... Imaging lens, 11A ... Imaging lens surface, 12, 112 ... 1st imaging device, 13 ... Micro lens array part, 13 '... Microlens, 13A ... refocusing surface, 113 ... pinhole, 14 ... second imaging element, 14A ... imaging surface, 15 ... image processing unit, 16 ... image sensor driving Means, 17 ... control part, 120 ... light receiving part, 121 ... transfer path, 122 ... light shielding part

Claims (9)

(A) an imaging lens,
(B) a first imaging element disposed on the focal plane of the imaging lens;
(C) a second image sensor, and
(D) imaging means;
With
Imaging is performed in the first imaging mode and the second imaging mode,
During imaging in the first imaging mode, light that has passed through the imaging lens forms an image on the imaging surface of the first imaging element,
In imaging in the second imaging mode, the light that has passed through the imaging lens passes through the first imaging element and reaches the imaging surface of the second imaging element by the imaging unit. apparatus.   The imaging apparatus according to claim 1, wherein the imaging unit includes a microlens array unit provided in the first imaging element.   The imaging apparatus according to claim 2, wherein the micro lens array unit faces the second imaging element. The second image sensor is composed of a plurality of light receiving elements,
The micro lens array part is composed of a plurality of micro lenses,
The imaging apparatus according to claim 2, wherein a plurality of light receiving elements constituting the second imaging element correspond to one microlens.   The imaging apparatus according to claim 1, wherein the image forming unit includes a plurality of pinholes provided in the first imaging element. The second image sensor is composed of a plurality of light receiving elements,
The imaging apparatus according to claim 5, wherein a plurality of light receiving elements constituting the second imaging element correspond to one pinhole.   The imaging apparatus according to claim 1, wherein the first imaging element allows only light having a specific wavelength to pass therethrough.   The imaging apparatus according to claim 7, wherein the second imaging element has sensitivity to a specific wavelength that has passed through the first imaging element. (A) an imaging lens,
(B) a first imaging element disposed on the focal plane of the imaging lens;
(C) a second image sensor, and
(D) imaging means;
An imaging method using an imaging apparatus comprising:
At the time of imaging in the first imaging mode, the light passing through the imaging lens is imaged on the imaging surface of the first imaging element,
In imaging in the second imaging mode, imaging that is characterized in that light that has passed through the imaging lens passes through the first imaging element and reaches the imaging surface of the second imaging element by the imaging means Method.

Images