JP2010263353A - Photo detecting device and image pickup device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a photo detecting device obtaining a color image without deteriorating effects of a lens array even when an optical system is equipped with the lens array composed of a microlens group, and to provide an image pickup device. <P>SOLUTION: The photo detecting device includes a lens array having a plurality of lenses arranged regularly on a single plane, and a photoelectric converter having a plurality of photoelectric conversion areas, each area comprising a plurality of pixels which are provided on the single plane to correspond with the irradiation ranges of transmitted light of the lenses in the lens array. The photoelectric conversion element has different spectral sensitivity characteristics in the first area centering on the optical axis of the lens, and the second area around the first region. The image pickup device equipped with the photo detecting device is also provided. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a light detection element and an imaging device.

  A conventional general digital still camera generates a color image signal by irradiating light collected by a focus lens to a CCD image sensor, a CMOS image sensor, or other imaging devices. Recently, however, an imaging apparatus having an optical system including a lens array including a group of microlenses arranged on a single plane between a lens and an imaging element has been proposed (for example, see Non-Patent Document 1). Such an imaging apparatus is called a plenoptic type imaging apparatus.

  Non-Patent Document 1 introduces an example of freely determining the depth of field by reconstructing an image obtained by such an optical system. Developments such as application and resolution improvement are considered. Moreover, as a technique using such an optical system, for example, Patent Documents 1 to 3 are cited.

JP 2003-163938 A JP 2004-146619 A JP 2007-317951 A

(Ren Ng) and 5 others “Light Field Photograph with a Hand-held Plenoptic Camera”, Stanford Tech Report CTSR 2005-02, p.1-11

  A digital still camera including an optical system including a lens array composed of such microlens groups can obtain information on a plurality of light beams separated for each microlens, so that control of depth of field and improvement of resolution can be obtained. It is expected to be usable for distance measurement using parallax. On the other hand, when a digital still camera is provided with an optical system including a lens array composed of such microlens groups, colorization of image data is essential.

  When a general imaging device generates a color image from a single-plate two-dimensional imaging device, as represented by the Bayer array, after obtaining color information by giving different spectral characteristics for each position, Full color image data is generated by determining a color for each pixel by interpolation processing.

  However, in the plenoptic type imaging device, there is a problem in that when a plurality of light flux information separated for each microlens is Bayered, the light flux information is lost. In addition, when trying to increase the light flux information separated by the Bayer arrangement, microlens miniaturization is required, and microlens processing miniaturization and imaging element miniaturization are required in accordance with microlens miniaturization requirements. There was also a problem that it was accompanied by technical difficulty and cost increase.

  Therefore, the present invention has been made in view of the above problems, and an object of the present invention is an effect obtained by providing the lens array even if the optical system includes a lens array including a microlens group. It is an object of the present invention to provide a novel and improved photodetecting element and imaging apparatus capable of obtaining a color image without impairing the above.

  In order to solve the above problems, according to one aspect of the present invention, a lens array in which a plurality of lenses provided on a single plane are regularly arranged, and an irradiation range of transmitted light of each lens of the lens array A photoelectric conversion unit including a plurality of photoelectric conversion elements provided on a single plane, wherein the photoelectric conversion element is provided with a first region provided around the optical axis of the lens, and the first There is provided a photodetecting element characterized by having different spectral sensitivity characteristics in the second region provided around the region. As a result, the main signal component is received in the first region used in the image reconstruction process, so that the loss of the separated light flux can be prevented from being thinned out from the original image, and the remaining signals in the second region are signaled. Multiple components can be realized by receiving the components.

  The first region has a spectral sensitivity characteristic corresponding to a luminance signal, and the second region has a spectral sensitivity characteristic corresponding to a color signal. As a result, the luminance component is received in the first region used in the image reconstruction process, so that the loss of the separated light flux can be prevented from being thinned out from the original image, and the color component can be added to the remaining second region. Colorization can be realized by receiving light.

  In each of the photoelectric conversion regions, the arrangement pattern of the spectral characteristics of the pixels is all the same. As a result, the design and processing for micro lens miniaturization can be made much easier than in the case where the arrangement pattern is irregular.

  The second region may be provided so as to be included in an irradiation range of transmitted light from each lens of the lens array. As a result, it is possible to avoid the occurrence of omission in the luminous flux information separated for each lens.

  As described above, according to the present invention, even if an optical system includes a lens array including a microlens group, a photodetecting element capable of obtaining a color image without impairing the effects of including the lens array. In addition, an imaging device can be provided.

It is explanatory drawing explaining an optical system provided with the lens array which consists of a micro lens group. It is explanatory drawing explaining an optical system provided with the lens array which consists of a micro lens group. It is explanatory drawing shown about the case where the image imaged with the optical system provided with a lens array is colorized. It is explanatory drawing shown about the case where the image imaged with the optical system provided with a lens array is colorized. It is explanatory drawing shown about the structure of the imaging device 100 concerning one Embodiment of this invention. It is explanatory drawing shown about the structure of the image sensor 106 used for the imaging device 100 concerning one Embodiment of this invention. It is explanatory drawing which shows what expanded the image sensor 106 shown in FIG. It is explanatory drawing which shows the relationship between the wavelength and spectrum intensity | strength of each color of R, G, and B which are three primary colors. It is explanatory drawing which shows the relationship between the wavelength of each color of cyan, magenta, and yellow and spectrum intensity. FIG. 10 is an explanatory diagram showing the relationship between the wavelength of the luminance signal and the spectrum intensity. It is a flowchart shown about the imaging method using the imaging device 100 concerning one Embodiment of this invention. 3 is a flowchart illustrating a method for generating a color image in the imaging apparatus 100 according to the embodiment of the present invention. It is explanatory drawing which shows the modification of a structure of the image sensor. It is explanatory drawing which shows the structure of microlens array 104 '. It is explanatory drawing shown about the structural example of the imaging sensor to which the primary color filter for obtaining the information of red, green, and blue was stuck. It is explanatory drawing shown about the structural example of the imaging sensor to which the primary color filter for obtaining the information of red, green, and blue was stuck. It is explanatory drawing which shows the structure of microlens array 104 ''.

  Exemplary embodiments of the present invention will be described below in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

  First, before describing a preferred embodiment of the present invention, an optical system having a lens array composed of a conventional microlens group will be described, and subsequently, problems of colorization when such an optical system is used will be described. explain.

  1 and 2 are explanatory diagrams for explaining an optical system including a lens array including a microlens group. FIG. 1 is an explanatory diagram showing a case where a lens array is provided between a lens for condensing light from a subject and an image pickup device when viewed from the side, and FIG. 2 shows an arrangement state of the lens array. It is explanatory drawing shown notionally.

  As shown in FIG. 1, in an optical system including a lens array composed of microlens groups, light from a subject passing through the main lens 11 is projected so as to be focused on each microlens 12 a of the lens array 12. . The image sensor 13 is irradiated with light transmitted through the microlens 12a.

  Non-Patent Document 1 discloses that the imaging sensor 13 is set with a diaphragm 14 of the main lens 11 so that light from adjacent microlenses 12a does not overlap. Non-Patent Document 1 introduces an example in which the depth of field is freely determined by reconstructing an image obtained by the optical system as shown in FIG. Accordingly, an optical system including a lens array including such a microlens group is considered to be developed for distance measurement using parallax, application to a three-dimensional image, and resolution improvement processing.

  When such an optical system is used for a general-purpose digital still camera, how to colorize a captured image becomes a problem. A method of performing spectroscopy using a dichroic mirror or the like and performing colorization using a plurality of imaging sensors is considered, but this is disadvantageous in terms of installation space and cost of the imaging sensor. A method of colorizing using is generally used. And when colorizing using a single-plate imaging sensor, there is a technique in which a spectral filter is pasted on the front surface of the light receiving element in a specific pattern arrangement and imaged, and later color information is aligned for every pixel by interpolation processing. Are known. However, if this method is applied as it is to an optical system including a lens array composed of microlens groups, a problem occurs.

  FIG. 3 is an explanatory diagram illustrating a case where an image captured by an optical system including a lens array is colorized by a spectral filter having a Bayer array that is widely used in digital still cameras. FIG. 3 shows a case in which a block of 8 vertical pixels × 8 horizontal pixels is associated with one microlens, and each circle shown in FIG. 3 represents a projection range of one microlens.

  Since data captured through a predetermined optical path is obtained as specific color information, it is appropriate to perform the same color for the reconstruction process. However, even if the spectral filter pattern shown in FIG. 3 has the largest amount of green (G) components, it can only have information in a checkered pattern, and therefore is in a defective state.

  In order to increase the information, a method of trying to colorize an image by changing the spectral characteristics in units of microlenses can be considered. FIG. 4 is an explanatory diagram illustrating a case where an image captured by an optical system including a lens array is colorized by a spectral filter having different characteristics in units of microlenses. 4 shows a case where a block of 8 vertical pixels × 8 horizontal pixels is associated with one microlens, as in FIG. 3, and each circle shown in FIG. 4 indicates the projection range of one microlens. Represents.

  When spectral filters having different spectral characteristics in units of microlenses as shown in FIG. 4 are used, the reconstruction processing for a captured image is performed as shown in FIG. It is also advantageous if a spectral filter having an array is used. However, since the optical system including the lens array further divides and captures information of one pixel, the number of pixels after reconstruction is reduced to the number of microlenses. Furthermore, since interpolation processing is executed from this reduced pixel, a digital still camera equipped with a lens array has a smaller number of pixels than a digital still camera having a normal optical system using the same imaging sensor. Will be forced. When a spectral filter as shown in FIG. 4 is used, 64 pixels are assigned to one microlens. Therefore, the number of recorded pixels is 1/64 of the number of pixels of the image sensor. Interpolation processing also has problems such as creating predictions of high-frequency components that are not recorded, and coloring borders due to phase shifts in color information, and if performed with a small number of pixels, degradation of image quality becomes more noticeable .

  Therefore, in one embodiment of the present invention described below, pixels that are imaged in an area used for image reconstruction within a range irradiated by each microlens receive a luminance signal, and complementary colors in other areas. An imaging sensor that obtains a color signal by attaching a filter is configured. Here, the region used for image reconstruction is a predetermined region centered on the optical axis of each microlens.

  FIG. 5 is an explanatory diagram showing the configuration of the imaging apparatus 100 according to the embodiment of the present invention. Hereinafter, the configuration of the imaging apparatus 100 according to the embodiment of the present invention will be described with reference to FIG.

  As shown in FIG. 5, the imaging apparatus 100 according to the embodiment of the present invention includes a main lens 102, a microlens array 104, an imaging sensor 106, a CPU 108, a memory 110, and an analog front end (AFE). And A / D converter 112, image input unit 114, color pixel generation unit 116, image reconstruction unit 118, digital back end (DBE) unit 120, image compression unit 122, and memory card driver 124. A display image generation unit 126, a display driver 128, a timing generator (TG) 130, a motor driver 132, and a focus lens motor 134.

  The main lens 102 includes a focus lens that is a lens for focusing on a subject, a zoom lens that changes a focal length, and the like. By moving the position of the focus lens included in the main lens 102 by driving the focus lens motor 134, the imaging apparatus 100 can focus on the subject.

  The microlens array 104 is a lens array composed of a plurality of microlens groups. The microlens array 104 is configured by regularly arranging microlenses 104a on a single plane. The light that has passed through the main lens 102 passes through each microlens of the microlens array 104 and is irradiated to the image sensor 106.

  The imaging sensor 106 generates an image signal from the light that has passed through each microlens 104 a that constitutes the microlens array 104. The imaging sensor 106 has a predetermined light receiving pattern corresponding to each microlens 104a, and as described above, pixels that are imaged in an area used for image reconstruction in the range irradiated by each microlens 104a. Is configured to receive a luminance signal, and a complementary color filter is attached in other areas to obtain a color signal. The configuration of the image sensor 106 will be described in detail later.

  The CPU 108 controls the operation of each unit of the imaging device 100. The CPU 108 can control the operation of each unit of the imaging apparatus 100 by sequentially executing computer programs stored in the imaging apparatus 100. The memory 110 stores information and data necessary for the operation of the imaging apparatus 100.

  The analog front end unit and A / D conversion unit 112 receives an analog signal photoelectrically converted by the image sensor 106, converts it into a digital signal, and outputs it. The signal converted into a digital signal by the analog front end unit and A / D conversion unit 112 is sent to the image input unit 114.

  The image input unit 114 receives the digital signal generated by the analog front end unit and the A / D conversion unit 112 and stores the digital signal in the memory 110. By storing the digital signal generated by the analog front end unit and the A / D conversion unit 112 in the memory 110, the imaging apparatus 100 can execute various signal processes on the digital signal.

  The color pixel generation unit 116 performs signal processing for generating color data for an image signal generated from light received by the image sensor 106. Specifically, the color pixel generation unit 116 generates color data for pixels in which no color information exists among the image signals generated by the image sensor 106. The color data generation processing in the color pixel generation unit 116 will be described in detail later.

  The image reconstruction unit 118 reconstructs an image captured through the microlens array 104. The image reconstruction unit 118 can change the subject to be focused by changing the depth of field by, for example, reconstruction of an image captured through the microlens array 104. In addition, the image reconstruction unit 118 may perform resolution improvement processing such as noise removal or color correction.

  The digital back-end unit 120 performs image processing on an image captured through the microlens array 104 and colorized by the color pixel generation unit 116. For example, the digital back-end unit 120 executes processing for enhancing saturation or image size. The process which converts is performed.

  The image compression unit 122 compresses image data into an appropriate format. The image compression format may be a reversible format or an irreversible format. As an example of a suitable format, you may convert into a JPEG (Joint Photographic Experts Group) format and a JPEG2000 format. The memory card driver 124 executes recording of the image data compressed by the image compression unit 122 to a memory card (not shown) and reading of the image data recorded on the memory card from the memory card. It is.

  The display image generation unit 126 generates an image (display image) to be displayed on a display unit (not shown) that displays a captured image and various setting screens of the imaging apparatus 100. For example, when a captured image is displayed on the display unit, the display image generation unit 126 generates a display image by compressing the image data in accordance with the resolution and screen size of the display unit. The display driver 128 executes processing for displaying the display image generated by the display image generation unit 126 on the display unit.

  The timing generator 130 inputs a timing signal to the image sensor 106. The shutter speed is determined by the timing signal from the timing generator 130. In other words, the drive of the image sensor 106 is controlled by the timing signal from the timing generator 130, and the image signal from the subject is incident within the time that the image sensor 106 is driven, thereby generating an electrical signal that is the basis of the image data. The

  The motor driver 132 drives the focus lens motor 134 based on the control of the CPU 108. The focus lens motor 134 controls the position of the main lens 102 by the motor. By controlling the position of the main lens 102 via the motor driver 132 and the focus lens motor 134, the focus of the subject can be adjusted.

  Although not shown in FIG. 5, the imaging apparatus 100 may include an aperture, a motor for adjusting the aperture, and a motor driver for driving the motor. Further, although not shown in FIG. 5, the imaging apparatus 100 includes a shutter button for starting a shooting operation, an operation button for setting shooting information such as an aperture, a shutter speed, and sensitivity. Also good.

  The configuration of the imaging device 100 according to the embodiment of the present invention has been described above. Next, the configuration of the imaging sensor 106 used in the imaging apparatus 100 according to an embodiment of the present invention will be described.

  FIG. 6 is an explanatory diagram illustrating the configuration of the imaging sensor 106 used in the imaging device 100 according to the embodiment of the present invention, and FIG. 7 is an explanatory diagram illustrating an enlarged version of the imaging sensor 106 illustrated in FIG. 6. It is.

  The circle shown in FIG. 6 shows the range irradiated with light that passes through one microlens 104a that constitutes the microlens array 104, like the circle shown in FIG. As shown in FIG. 6, a plurality of pixels are assigned to the imaging sensor 106 corresponding to a range irradiated with light that passes through one microlens 104 a. In the example shown in FIGS. 6 and 7, a total of 64 pixels of 8 pixels in the vertical direction and 8 pixels in the horizontal direction are assigned to one microlens 104a, and the light transmitted through one microlens 104a is the 64 pieces of light. Photoelectric conversion is performed by the pixel.

  In the imaging sensor 106 shown in FIGS. 6 and 7, 64 pixels allocated to one microlens 104a include an area including a pixel that obtains a luminance signal, and an area including a pixel that obtains a complementary color signal. It is divided into. An area used for the reconstruction process in the image reconstruction unit 118, that is, an area including pixels captured near the optical axis of each microlens is an area for obtaining a luminance signal, and an area including surrounding pixels is a complementary color. This is the area from which signals are obtained. In the imaging sensor 106 shown in FIG. 6, an area composed of pixels indicated by Y is an area for obtaining a luminance signal, and an area composed of pixels indicated by Cy, Mg, and Ye is an area for obtaining a complementary color signal. Pixels indicated by Cy, Mg, and Ye are pixels that obtain cyan, magenta, and yellow information, respectively.

  FIG. 7 is an explanatory diagram showing an enlarged view of 64 pixels assigned to one microlens 104a. The configuration of the image sensor 106 will be described in more detail with reference to FIG.

  An area used for image reconstruction corresponds to an area composed of pixels indicated by Y0 to Y51 among the pixels shown in FIG. Complementary color filters are attached to the pixels in the other regions, respectively, so that information on cyan, magenta, and yellow can be obtained. Pixels for obtaining cyan information are pixels indicated by Cy0 to Cy3 in FIG. 7, pixels for obtaining magenta information are pixels indicated by Mg0 to Mg3 in FIG. 7, and pixels for obtaining yellow information are: These are the pixels indicated by Ye0 to Ye3 in FIG. An RGB signal can be obtained by referring to information on cyan, magenta, and yellow from a luminance signal obtained by irradiating the pixels Y0 to Y51 with light.

  In a pixel that has received a complementary color signal, the light flux information is different from that in a region that receives the luminance signal. However, it is known that the complementary color signal has a lower human sensitivity than the luminance signal. Therefore, in the present embodiment, the RGB signal at the position where the luminance signal is received is calculated by applying a weight according to the distance difference between the position where the luminance signal of the imaging sensor 106 is received and each light receiving position of the plurality of complementary color signals. Generate.

  FIG. 8 is an explanatory diagram showing the relationship between the wavelength of each of the three primary colors R, G, and B and the spectrum intensity, and FIG. 9 shows the relationship between the wavelength of each color of cyan, magenta, and yellow and the spectrum intensity. FIG. 10 is an explanatory diagram showing the relationship between the wavelength of the luminance signal and the spectrum intensity. As shown in FIGS. 8 to 10, the relationship between the wavelength of the luminance signal and the spectrum intensity has characteristics including the relationship between the wavelength of each color of cyan, magenta, and yellow and the spectrum intensity shown in FIG. is doing. Therefore, it is possible to generate an RGB signal at the position where the luminance signal is received by calculating the weighting according to the distance difference between each light receiving position of the plurality of complementary color signals.

  The configuration of the imaging sensor 106 used in the imaging device 100 has been described above. Next, an imaging method using the imaging device 100 according to an embodiment of the present invention and a color image generation method will be described.

  FIG. 11 is a flowchart showing an imaging method using the imaging apparatus 100 according to an embodiment of the present invention, and FIG. 12 is a flowchart showing a color image generation method in the imaging apparatus 100 according to an embodiment of the present invention. It is. Hereinafter, an imaging method using the imaging apparatus 100 according to the embodiment of the present disclosure and a color image generation method will be described with reference to FIGS. 11 and 12.

  When a subject is photographed using the imaging apparatus 100 according to the embodiment of the present invention, an optimum aperture value is first set automatically using the photometric result of the subject or by the photographer's hand (step S101). Subsequently, the optimum shutter speed for photographing the subject is set automatically using the photometric result of the subject or by the photographer's hand, and the gain for photographing the image is determined (step S102). . Then, the main subject is focused on the microlens array 104 by the motor driver 132 and the focus lens motor 134 moving the position of the focus lens (step S103).

  When the main subject is focused on the microlens array 104, the photographing process in the imaging device 100 is executed by pressing the shutter button (step S104). The imaging process in the imaging apparatus 100 is executed by irradiating the imaging sensor 106 with image light from a subject. The timing generator 130 controls the irradiation of light to the image sensor 106 so that the irradiation is performed only during the shutter speed period set in step S102.

  When image light from the subject passes through the main lens 102 and the microlens array 104 and is irradiated to the image sensor 106, photoelectric conversion is performed by the image sensor 106 to generate an electrical signal. The electrical signal generated by the image sensor 106 is converted into a digital signal by the analog front end unit and the A / D conversion unit 112, and the converted digital signal is stored in the memory 110 as image data by the image input unit 114. (Step S105).

  When the image data is stored in the memory 110, the color pixel generation unit 116 reads the image data stored in the memory 110, and generates an RGB image for each region divided by the microlens 104a (step S106). The RGB image generation processing by the color pixel generation unit 116 in step S106 will be described in detail later.

  When the RGB image generation processing by the color pixel generation unit 116 in step S106 is completed, the image reconstruction unit 118 subsequently acquires reconstruction parameters used for the image reconstruction processing (step S107). The reconstruction parameters used in the image reconstruction process may include information such as distance information from the imaging apparatus 100 to the subject and the pitch between the lenses of the microlens 104a constituting the microlens array 104, for example. Good.

  When the acquisition of reconstruction parameters by the image reconstruction unit 118 in step S107 is completed, the image reconstruction unit 118 performs image data reconstruction processing using the acquired parameters (step S108). ). By reconstructing the image data by the image reconstruction unit 118, it is possible to generate an image focused on a subject different from that at the time of shooting. Note that the image data reconstruction processing is disclosed in, for example, Non-Patent Document 1, and detailed description thereof is omitted here.

  When the image reconstruction process by the image reconstruction unit 118 in step S108 is completed, the digital backend unit 120 subsequently executes various image processes on the reconstructed image data (step S109). The various image processings here may include, for example, noise removal processing, saturation enhancement processing, image size conversion processing, and the like. Image data that has undergone image processing is stored in the memory 110.

  When the various types of image processing by the digital back end unit 120 in step S109 are completed, the image compression unit 122 subsequently performs compression processing on the image data on which image processing has been performed (step S110). When the compression process for the image data is completed, the memory card driver 124 stores the compressed image data in the recording medium (step S111).

  The imaging method using the imaging device 100 according to the embodiment of the present invention has been described above. Next, the RGB image generation processing by the color pixel generation unit 116 shown in step S106 of FIG. 11 will be described in detail.

  FIG. 12 is a flowchart for explaining the RGB image generation processing by the color pixel generation unit 116 shown in step S <b> 106 of FIG. 11. Here, as shown in FIGS. 6 and 7, a case where a region corresponding to one microlens 104a is an area of 64 pixels in total of 8 pixels vertically × 8 pixels horizontally will be described as an example.

  First, the color pixel generation unit 116 sets a work variable k representing the number of microlenses 104a constituting the microlens array 104 to 0 (step S121). When the variable k is set to 0 in step S121, the color pixel generation unit 116 subsequently replaces the area of 64 pixels in total of 8 pixels by 8 pixels by the matrix of 8 rows by 8 columns. A variable n representing an element is set to 0 (step S122), and then a variable m representing an element of the matrix column is set to 0 (step S123).

  When each variable is set to 0, the color pixel generation unit 116 then sets cyan (Cy [n] [m]) and magenta (Mg [n] [M] [n] in the region divided by the above-described 64-pixel region. m]) and yellow (Ye [n] [m]) are calculated (step S124). Cy [n] [m], Mg [n] [m], and Ye [n] [m] are respectively calculated by weighted averages corresponding to the distances from the four corner pixels of the 64-pixel region. First, the distance between the calculation target pixel and each pixel at the four corners is calculated. The distances d0, d1, d2, and d3 from the pixels at the four corners to the pixel to be calculated can be expressed by the following formulas 1 to 4, respectively. As shown in FIG. 7, since the positions of the Cy, Mg, and Ye pixels at the four corners are different, the distances d0 from the positions of the cyan, magenta, and yellow pixels at the four corners to the pixel to be calculated. d1, d2, and d3 are different for cyan, magenta, and yellow, respectively. Accordingly, it is appropriate that the following formulas 1 to 4 are represented by different formulas for cyan, magenta, and yellow, but here, only the concept is shown, and in the following, for simplification of explanation, cyan, magenta The distinction between d0, d1, d2, and d3 for each pixel of yellow is omitted.

  When the color pixel generation unit 116 calculates the distances d0, d1, d2, and d3 from the pixels at the four corners to the calculation target pixel, the color pixel generation unit 116 continues to calculate the distances from the four corner pixels obtained by the above Equations 1 to 4. The sum d of the distances d0, d1, d2, d3 to the pixel to be calculated is obtained by the following formula 5.

  When the sum of the distances from the pixels at the four corners to the pixel to be calculated is calculated, Cy [n] [m], Mg [n] [m], and Ye [n] [m] at the pixel can be obtained. Cy [n] [m], Mg [n] [m], and Ye [n] [m] are the distances d0, d1, d2, and d3 from the pixels at the four corners calculated in Equation 5 to the pixel to be calculated. Using the sum d, it can be obtained by the following formulas 6 to 8. In Equations 6 to 8 below, Cy0, Cy1, Cy2, and Cy3 are cyan values in the pixels Cy0, Cy1, Cy2, and Cy3 shown in FIG. 7, and Mg0, Mg1, Mg2, and Mg3 are 7 are magenta values in the pixels of Mg0, Mg1, Mg2, and Mg3, and Ye0, Ye1, Ye2, and Ye3 are yellow values in the pixels of Ye0, Ye1, Ye2, and Ye3 shown in FIG.

  When cyan (Cy [n] [m]), magenta (Mg [n] [m]), and yellow (Ye [n] [m]) are calculated in this way, the color pixel generation unit 116 then performs the above-described process. Using the luminance signal Y [k] [n] [m] in the area divided by the 64-pixel area, R [n] [m] and G [n] [m] in the pixels in the divided area , B [n] [m] are obtained (step S125). Since R, G, and B are complementary colors of Cy, Mg, and Ye, the R, G, and B values of the pixel are subtracted from the luminance signal Y (Y [n] [m]) of each pixel. Can be derived. R [n] [m], G [n] [m], and B [n] [m] can be obtained by the following formulas 9 to 11.

  When R [n] [m], G [n] [m], and B [n] [m] are calculated in step S125, the color pixel generation unit 116 subsequently increments the value of m by one (step S126). ). When the value of m is incremented by 1, the color pixel generation unit 116 subsequently determines whether the value of m is less than 8 (step S127). If the value of m is less than 8, the process returns to step S124. On the other hand, if the value of m is 8 or more, the color pixel generation unit 116 subsequently increments the value of n by 1 (step S128). When the value of n is incremented by 1, the color pixel generation unit 116 subsequently determines whether or not the value of n is less than 8 (step S129). If the value of n is less than 8, the process returns to step S123 to reset the value of m. On the other hand, if the value of n is 8 or more, this means that the calculation of the R, G, and B values has been completed for all 64 pixels assigned to one microlens 104a. Next, the color pixel generation unit 116 increments the value of k by one (step S130). When the value of k is incremented by 1, the color pixel generation unit 116 subsequently determines whether or not the value of k is less than the number of microlenses 104a constituting the microlens array 104 (step S131). If the value of k is less than the number of microlenses 104a, the process returns to step S122 to reset the value of n. On the other hand, if the value of k is equal to or greater than the number of microlenses 104a, this means that the calculation of the R, G, and B values for all the microlenses 104a has been completed, and thus the series of processing ends.

  In Equations 6 to 8, Cy [n] [m], Mg [n] [m], and Ye [n] [m] are simply calculated by a weighted average according to the distance from the four corners. The present invention is not limited to such an example. In the example shown in FIGS. 6 and 7, since the complementary color portions that acquire cyan, magenta, and yellow information are located outside the circle, the light that has passed through the microlens 104a is sufficiently irradiated to these complementary color portions. There are cases where this is not done. Therefore, when the light amount of the complementary color portion is insufficient, the right side portion that is averaged may be changed in accordance with the actual light amount in Equations 6 to 8. For example, Cy [n] [m], Mg [n] [m], and Ye [n] [m] may be calculated by multiplying by a predetermined coefficient α as shown in Equations 12 to 14 below. Good.

  In the formulas 12 to 14, the same coefficient α is used for Cy [n] [m], Mg [n] [m], and Ye [n] [m], but the present invention is limited to this example. It goes without saying that it is not done. Separate coefficients may be used for cyan, magenta, and yellow, and any two of cyan, magenta, and yellow may use different coefficients from the other one.

  Further, in order to compensate for the insufficient light amount of the complementary color portion, for example, as shown in FIG. 13, the image sensor 106 is configured so that the complementary color portion is located inside the region irradiated with the light transmitted through the microlens 104a. Also good. In the example shown in FIG. 13, pixels that obtain cyan information are pixels indicated by Cy0 to Cy3, pixels that obtain magenta information are pixels indicated by Mg0 to Mg3, and pixels that obtain yellow information are Ye0. This is a pixel indicated by ~ Ye3. Further, from the luminance signal obtained by irradiating the region composed of pixels Y0 to Y39 in FIG. 13 with the light transmitted through the microlens 104a, the microlens 104a is applied to the pixels Cy0 to Cy3, Mg0 to Mg3, and Ye0 to Ye3. RGB signals can be obtained by referring to the information of cyan, magenta, and yellow obtained by irradiating the light that has passed through.

  Further, in order to compensate for the insufficient light amount of the complementary color portion, a microlens array 104 ′ in which rectangular microlenses 104a ′ are regularly arranged as shown in FIG. 14, for example, is used instead of the microlens array 104. May be. By using the rectangular microlens 104a ′, the irradiation range of the transmitted light is widened, and the shortage of light in the complementary color portion can be compensated.

  The RGB image generation processing by the color pixel generation unit 116 has been described above.

  Although the case where the image sensor 106 to which the complementary color filter for obtaining cyan, magenta, and yellow information is attached has been described so far, the present invention is not limited to such an example. The present invention may be applied to an image sensor to which primary color filters for obtaining red, green, and blue information are attached. FIG. 15 and FIG. 16 are explanatory diagrams showing a configuration example of an image sensor to which primary color filters for obtaining red, green, and blue information are attached. The example shown in FIG. 15 is for the case where it has the same configuration as the imaging sensor to which the complementary color filter shown in FIG. 7 is attached. The pixels used for image reconstruction correspond to those indicated by Y0 to Y51 among the pixels shown in FIG. A primary color filter is affixed to each of the pixels in the other regions so as to obtain red, green, and blue information. The values of R [n] [m], G [n] [m], and B [n] [m] of each pixel in FIG. 15 can be calculated by the following equations, respectively. Note that the calculation formulas for the distances d0, d1, d2, d3 from the pixels at the four corners to the calculation target pixel and the sum d of d0, d1, d2, d3 are the same as those in the above formulas 1-5.

  On the other hand, FIG. 16 has the same configuration as the image sensor 106 shown in FIG. 13 in which the complementary color portion is positioned inside the region irradiated with the light transmitted through the microlens 104a. As shown in FIG. 16, the imaging sensor 106 may be configured such that primary color portions that acquire red, green, and blue information are located inside the region irradiated with light transmitted through the microlens 104 a. .

  As described above, according to an embodiment of the present invention, when a color image signal is generated from light transmitted through the microlens array 104, a plurality of pixels of the image sensor 106 are assigned to one microlens 104a. Provide a separate area. The region is configured such that a region around the optical axis of the microlens 104a obtains a luminance signal, and a peripheral region of the region obtains a complementary color signal or a primary color signal. When generating a color image signal, complementary color data or primary color data of each pixel in the region is calculated by a weighted average according to the distance from the four corner pixels of the region, and the calculated complementary color data of each pixel Alternatively, the color information of each pixel is calculated using the primary color data and the luminance data of the pixel.

  As a result, according to an embodiment of the present invention, luminance interpolation processing is not performed using peripheral pixels, so that there is no reduction in resolution or unnatural error patterns in the peripheral portion, and weighting from a wide range. In order to refer to the average processing value, it is possible to reduce the occurrence of false colors due to phase shift compared to the case where Bayer interpolation used in a conventional digital still camera is used.

  The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field to which the present invention pertains can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that these also belong to the technical scope of the present invention.

  For example, in the above-described embodiment, the imaging sensor 106 is configured to obtain a luminance signal in the region centered on the optical axis of the microlens 104a and a complementary color signal or primary color signal in the peripheral region of the region. The present invention is not limited to such an example. In contrast to the above-described embodiment, the imaging sensor may be configured to obtain, for example, a complementary color signal or a primary color signal in an area centered on the optical axis of the microlens 104a, and a luminance signal in the peripheral area of the area. .

  For example, in the above-described embodiment, the microlens array 104 has a configuration in which the microlenses 104a are arranged in a grid pattern, but the present invention is not limited to such an example. The microlenses constituting the microlens array may be arranged in, for example, a honeycomb shape other than the lattice shape. FIG. 17 is an explanatory diagram showing a microlens array 104 ″ in which microlenses 104a ″ are regularly arranged in a honeycomb shape. In FIG. 17, the shape of one microlens 104a ′ constituting the microlens array 104 ″ having the honeycomb structure is circular, but in the present invention, one microlens constituting the microlens array having the honeycomb structure. The shape is not limited to such an example.

  The present invention can be applied to a light detection element and an image pickup apparatus, and in particular, can be applied to a light detection element that detects transmitted light of a lens array including a plurality of lenses and an image pickup apparatus using the light detection element.

DESCRIPTION OF SYMBOLS 100 Imaging device 102 Main lens 104,104 ', 104''Microlens array 104a, 104a', 104a '' Microlens 106 Imaging sensor 108 CPU
110 Memory 112 Analog Front End Unit and A / D Converter 114 Image Input Unit 116 Color Pixel Generation Unit 118 Image Reconstruction Unit 120 Digital Back End Unit 122 Image Compression Unit 124 Memory Card Driver 126 Display Image Generation Unit 128 Display Driver 130 Timing Generator 132 Motor driver 134 Focus lens motor

Claims (4)

A lens array in which a plurality of lenses provided on a single plane are regularly arranged;
A photoelectric conversion unit including a plurality of photoelectric conversion regions including a plurality of pixels provided on a single plane corresponding to an irradiation range of transmitted light from each lens of the lens array;
With
Each of the photoelectric conversion regions has different spectral sensitivity characteristics in a first region provided around the optical axis of the lens and a second region provided around the first region. Detection element.   The light detection element according to claim 1, wherein the first region has a spectral sensitivity characteristic corresponding to a luminance signal, and the second region has a spectral sensitivity characteristic corresponding to a color signal.   3. The photodetector according to claim 1, wherein each of the photoelectric conversion regions has the same arrangement pattern of spectral characteristics of the pixels. 4. The light detection element according to claim 1, wherein the second region is provided so as to be included in an irradiation range of transmitted light from each lens of the lens array.

Images