JP2013175807A - Image pickup device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve the problem that complicated photographing optics needs to be prepared.SOLUTION: An image pickup device comprises: a spectral unit which separates incident light into first and second spectral components; a first image sensor including a first photoelectric conversion element group having a plurality of photoelectric conversion elements for receiving the first spectral component and converting it into a first electric signal arranged in two-dimensional form, which includes, in at least part of the plurality of photoelectric conversion elements, at least one of a camera filter and an opening formed at a position where light flux from a specific partial region in the cross sectional region of the incident light is transmitted; and a second image sensor including a second photoelectric conversion element group having a plurality of photoelectric conversion elements for receiving the second spectral component and converting it into a second electric signal arranged in two-dimensional form, which includes, in at least part of the plurality of photoelectric conversion elements, at least one of a camera filter and an opening formed at a position where light flux from a specific partial region in the cross sectional region of the incident light is transmitted. A set of the photoelectric conversion elements of the first image sensor on which the first spectral component from an object point is incident and the photoelectric conversion elements of the second image sensor on which the second spectral component from the same object point is incident differs at least partly from each other in the presence of a color filter and in at least the wavelength characteristic of the color filter, the presence of the opening, and the position of the opening.

Description

  The present invention relates to an imaging apparatus.

A stereo imaging device that captures a stereo image composed of a right-eye image and a left-eye image using two imaging optical systems is known. Such a stereo imaging device causes parallax to occur in two images obtained by imaging the same subject by arranging two imaging optical systems at regular intervals.
[Prior art documents]
[Patent Literature]
[Patent Document 1] JP-A-8-47001

  In order to acquire parallax images, each of which is a color image, it is necessary to prepare a complicated photographing optical system for capturing each parallax image.

  In the first aspect of the present invention, a spectroscopic unit that splits incident light from one optical system into a first spectrum and a second spectrum, and receives the first spectrum and photoelectrically converts it into a first electric signal. A plurality of photoelectric conversion elements are arranged in a two-dimensional array, and at least part of the plurality of photoelectric conversion elements is provided with a color filter and a cross-sectional area of the incident light. A first imaging element provided with at least one of openings provided at positions where light beams from a specific partial region pass, and a plurality of photoelectric conversions that receive the second spectrum and photoelectrically convert it into a second electrical signal. A second photoelectric conversion element group in which the elements are two-dimensionally arranged; for at least a part of the plurality of photoelectric conversion elements, from a color filter and a specific partial region in a cross-sectional region of the incident light; An opening provided at a position through which the luminous flux passes. A second image sensor provided with at least one of the photoelectric conversion element of the first image sensor on which the first spectrum from the same object point is incident and the photoelectric conversion element of the second image sensor on which the second spectrum is incident Provided is an imaging device in which at least one part of the set differs in at least one of presence / absence of the color filter, wavelength characteristics of the color filter, presence / absence of the opening, and position of the opening.

  It should be noted that the above summary of the invention does not enumerate all the necessary features of the present invention. In addition, a sub-combination of these feature groups can also be an invention.

It is a figure explaining the structure of the digital camera 10 which concerns on embodiment of this invention. It is the schematic showing the cross section of the image pick-up element which concerns on embodiment of this invention. 2 is a schematic diagram illustrating a state in which a part of the first image sensor 100 is enlarged. FIG. It is a conceptual diagram explaining the relationship between a parallax pixel and a to-be-photographed object. It is a conceptual diagram explaining the process which produces | generates a parallax image. It is a figure which shows the other example of the repeating pattern 110. FIG. It is a figure which shows the example of the two-dimensional repeating pattern 110. FIG. It is a figure explaining the other shape of the opening part 104. FIG. It is a figure explaining a Bayer arrangement. It is a figure explaining the imaging process of the parallax image which has the 1st image sensor 100 and the 2nd image sensor 300. FIG. It is a figure explaining the imaging process of the parallax image which has another 1st image sensor 100 and the 2nd image sensor 300. FIG. It is a figure explaining another 1st image sensor 100 and the 2nd image sensor 300. FIG. FIG. 6 is a partial configuration diagram of a digital camera imaging optical unit 610 in which a spectroscopic unit 650 splits a subject light beam into three. It is a front view explaining the 1st image sensor 100, the 2nd image sensor 300, and the 3rd image sensor 400 in the digital camera image pick-up optical part 610 of FIG. FIG. 14 is a front view illustrating another first image sensor 100, second image sensor 300, and third image sensor 400 in the digital camera imaging optical unit 610 of FIG. FIG. 6 is a partial configuration diagram of a digital camera imaging optical unit 710 in which a spectroscopic unit 750 splits a subject light beam into four. It is a front view explaining the 1st image sensor 100, the 2nd image sensor 300, the 3rd image sensor 400, and the 4th image sensor 500 in the digital camera image pick-up optical part 710 of FIG. FIG. 17 is a front view illustrating another first image sensor 100, second image sensor 300, third image sensor 400, and fourth image sensor 500 in the digital camera imaging optical unit 710 of FIG. FIG. 17 is a front view illustrating another first image sensor 100, second image sensor 300, third image sensor 400, and fourth image sensor 500 in the digital camera imaging optical unit 710 of FIG. FIG. 17 is a front view illustrating another first image sensor 100, second image sensor 300, third image sensor 400, and fourth image sensor 500 in the digital camera imaging optical unit 710 of FIG. It is a front view explaining the image pick-up element which changed the magnitude | size of opening. It is a front view explaining another image pick-up element which changed the magnitude | size of opening. It is a table | surface explaining the kind of image pick-up element. It is a table | surface explaining the kind of spectroscopic part. It is a table | surface explaining the variation of the combination in the case of providing two image sensors. It is a table | surface explaining the variation of the combination in the case of providing three image sensors.

  Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, not all the combinations of features described in the embodiments are essential for the solving means of the invention.

  The digital camera according to the present embodiment, which is an embodiment of the imaging apparatus, generates images with a plurality of viewpoints by shooting once and stores them as a RAW image data set. Each image having a different viewpoint is called a parallax image.

  FIG. 1 is a diagram illustrating the configuration of a digital camera 10 according to an embodiment of the present invention. The digital camera 10 includes a photographic lens 20 as a photographic optical system which is an example of one optical system, and a subject light beam which is an example of incident light incident along the optical axis 21 is captured by the first image sensor 100 or the second image sensor. Lead to element 300. The photographing lens 20 may be an interchangeable lens that can be attached to and detached from the digital camera 10. The digital camera 10 includes a spectroscopic unit 50, a first image sensor 100, a control unit 201, an A / D conversion circuit 202, a memory 203, a drive unit 204, an image processing unit 205, a memory card IF 207, an operation unit 208, a display unit 209, An LCD drive circuit 210, an AF sensor 211, a storage control unit 238, a second image sensor 300, an A / D conversion circuit 302, a memory 303, and a drive unit 304 are provided.

  As shown in the figure, the direction parallel to the optical axis 21 toward the first image sensor 100 is defined as the z-axis plus direction, and the direction toward the front of the page in the plane orthogonal to the z-axis is the x-axis plus direction and the upward direction on the page. Is defined as the y-axis plus direction. In relation to the composition in photographing, the x-axis is the horizontal direction and the y-axis is the vertical direction. In the following several figures, the coordinate axes are displayed so that the orientation of each figure can be understood with reference to the coordinate axes of FIG.

  The taking lens 20 is composed of a plurality of optical lens groups, and forms an image of a subject light beam in the vicinity of its focal plane. In FIG. 1, for convenience of explanation, the photographic lens 20 is represented by a single virtual lens arranged in the vicinity of the pupil. The spectroscopic unit 50 splits the subject light flux from the photographing lens 20 into the first spectroscopic DL1 and the second spectroscopic DL2. The first light split DL1 proceeds to the first image sensor 100 after being split. The second light split DL2 travels to the second image sensor 300 after being split. Thereby, a parallax image can be picked up by the taking lens 20 without providing a complicated optical system. For example, the spectroscopic unit 50 can be configured by a half mirror such as a pellicle mirror that splits a light beam based on a set light amount, a spectroscopic mirror such as a dichroic mirror that splits a light beam based on a wavelength band, and a dichroic prism. When the spectroscopic unit 50 is configured by a half mirror, the spectroscopic unit 50 transmits a part of the subject light beam as the first spectroscopic DL1, and reflects the rest as the second spectroscopic DL2. As a result, the spectroscopic unit 50 separates the subject light flux with a predetermined light amount ratio. An example of the predetermined light amount ratio is 1: 1. In addition, another example of the predetermined light amount ratio is an arbitrary ratio that can correspond to a setting that widens the application range of white balance by variations of the first image sensor 100 and the second image sensor 300. When the spectroscopic unit 50 is configured by a dichroic mirror or a dichroic prism, the spectroscopic unit 50 transmits light in the first wavelength band of the subject light beam as the first spectral DL1, and has a second wavelength different from the first wavelength band. The light in the band is reflected as the second spectral DL2. An example of the first wavelength band is a visible light region of 400 nm to 800 nm. An example of the second wavelength band is infrared light from 800 nm to 1 mm.

  The first image sensor 100 is disposed near the focal plane of the photographic lens 20. The first imaging element 100 receives a first spectral DL1, and includes a photoelectric conversion element group in which a plurality of photoelectric conversion elements that are photoelectrically converted into a first image signal that is an example of a first electric signal are two-dimensionally arranged. Have. The first image sensor 100 is an image sensor such as a CCD or CMOS sensor. In the first image sensor 100, a color filter is provided for at least one of the photoelectric conversion elements. The second imaging element 300 receives a second spectral DL2 and generates a photoelectric conversion element group in which a plurality of photoelectric conversion elements that are photoelectrically converted into a second image signal that is an example of a second electric signal are two-dimensionally arranged. Have. The second image sensor 300 is an image sensor such as a CCD or CMOS sensor. The second image sensor 300 is provided with an aperture mask that is disposed at a position that allows the subject luminous flux from a specific partial area in the cross-sectional area to pass through at least one of the photoelectric conversion elements. The first image sensor 100 and the second image sensor 300 are different from each other in at least one of the wavelength characteristics of the color filter and the position of the aperture mask at corresponding positions in the array. The first image sensor 100 is timing-controlled by the drive unit 204, converts the subject image formed on the light receiving surface into an image signal, and outputs the image signal to the A / D conversion circuit 202.

  The A / D conversion circuit 202 converts the image signal output from the first image sensor 100 into a digital signal and outputs the digital signal to the memory 203 as RAW original image data. The image processing unit 205 performs various image processing using the memory 203 as a work space, and generates image data. The image processing unit 205 includes an image extracting unit 231, an image complementing unit 232, and an image combining unit 233. The image extraction unit 231 uses the electrical signals of the photoelectric conversion element group of the first image sensor 100 stored in the memory 203 and the electrical signals of the photoelectric conversion element group of the second image sensor 300 stored in the memory 303 to output a plurality of pixels. Color data and parallax data of a plurality of pixels are extracted. Based on the extracted color data and parallax data, the image complementing unit 232 supplements the color data and the parallax data to pixels that do not have color data and parallax data. The image synthesis unit 233 synthesizes the complemented pixel color data and the corresponding pixel parallax data, and stores them in the memory card 220. Details of each processing will be described later. The A / D conversion circuit 302, the memory 303, and the drive unit 304 have the same configuration as the A / D conversion circuit 202, the memory 203, and the drive unit 204, respectively.

  The image processing unit 205 also has general image processing functions such as adjusting image data according to the selected image format. The generated image data is converted into a display signal by the LCD drive circuit 210 and displayed on the display unit 209. The various image data are recorded on the memory card 220 mounted on the memory card IF 207 by the storage control unit 238.

  The AF sensor 211 is a phase difference sensor in which a plurality of distance measuring points are set for the subject space, and detects the defocus amount of the subject image at each distance measuring point. A series of shooting sequences is started when the operation unit 208 receives a user operation and outputs an operation signal to the control unit 201. Various operations such as AF and AE accompanying the imaging sequence are executed under the control of the control unit 201. For example, the control unit 201 analyzes the detection signal of the AF sensor 211 and executes focus control for moving a focus lens that constitutes a part of the photographing lens 20. In addition, you may comprise so that the parallax pixel mentioned later may combine the function of AF sensor 211. FIG. In this case, the AF sensor 211 can be omitted.

  Next, the configuration of the first image sensor 100 will be described in detail. FIG. 2 is a schematic diagram showing a cross section of the image sensor according to the embodiment of the present invention. FIG. 2A is a schematic cross-sectional view of the first image sensor 100 in which the color filter 102 and the aperture mask 103 are configured separately. FIG. 2B is a schematic cross-sectional view of an image sensor 120 including a screen filter 121 in which a color filter portion 122 and an opening mask portion 123 are integrally formed as a modification of the first image sensor 100.

  As shown in FIG. 2A, the first image sensor 100 is configured by arranging a micro lens 101, a color filter 102, an aperture mask 103, a wiring layer 105, and a photoelectric conversion element 108 in order from the subject side. . The photoelectric conversion element 108 is configured by a photodiode that converts incident light into an electrical signal. A plurality of photoelectric conversion elements 108 are two-dimensionally arranged on the surface of the substrate 109.

  An image signal converted by the photoelectric conversion element 108, a control signal for controlling the photoelectric conversion element 108, and the like are transmitted and received through the wiring 106 provided in the wiring layer 105. In addition, an opening mask 103 having openings 104 provided in one-to-one correspondence with each photoelectric conversion element 108 is provided in contact with the wiring layer. As will be described later, the opening 104 is shifted for each corresponding photoelectric conversion element 108 so that the relative position is precisely determined. As will be described in detail later, parallax occurs in the subject light beam received by the photoelectric conversion element 108 by the action of the opening mask 103 including the opening 104.

  On the other hand, the aperture mask 103 does not exist on the photoelectric conversion element 108 that does not generate parallax. In other words, it can be said that the aperture mask 103 having the aperture 104 that does not limit the subject luminous flux incident on the corresponding photoelectric conversion element 108, that is, allows the entire effective luminous flux to pass therethrough is provided. An aperture mask that does not cause parallax, but substantially defines the subject luminous flux that is incident by the opening 107 formed by the wiring 106, and allows the wiring 106 to pass through the entire effective luminous flux that does not cause parallax. It can also be captured. Note that the opening 107 may be formed in the upper wiring 106 of the wiring layer 105. The opening mask 103 may be arranged separately and independently corresponding to each photoelectric conversion element 108, or may be formed collectively for a plurality of photoelectric conversion elements 108 in the same manner as the manufacturing process of the color filter 102. .

  The color filter 102 is provided on the opening mask 103. The color filter 102 is a filter provided in a one-to-one correspondence with each photoelectric conversion element 108, which is colored so as to transmit a specific wavelength band to each photoelectric conversion element 108. In order to output a color image, it is only necessary to arrange at least three different color filters. These color filters can be said to be primary color filters for generating a color image. The primary color filter combination is, for example, a red filter that transmits the red wavelength band, a green filter that transmits the green wavelength band, and a blue filter that transmits the blue wavelength band. As will be described later, these color filters are arranged in a lattice pattern corresponding to the photoelectric conversion elements 108. The color filter may be not only a combination of primary colors RGB, but also a combination of YCMg complementary color filters.

  The microlens 101 is provided on the color filter 102. The microlens 101 is a condensing lens for guiding more incident subject light flux to the photoelectric conversion element 108. The microlenses 101 are provided in a one-to-one correspondence with the photoelectric conversion elements 108. In consideration of the relative positional relationship between the pupil center of the taking lens 20 and the photoelectric conversion element 108, the optical axis of the microlens 101 is shifted so that more subject light flux is guided to the photoelectric conversion element 108. It is preferable. Furthermore, the arrangement position may be adjusted so that more specific subject light beam, which will be described later, is incident along with the position of the opening 104 of the opening mask 103.

  As described above, one unit of the aperture mask 103, the color filter 102, and the microlens 101 provided in one-to-one correspondence with each photoelectric conversion element 108 is referred to as a pixel. The pixels are arranged continuously and periodically. In particular, a pixel provided with the opening mask 103 that generates parallax is referred to as a parallax pixel, and a pixel that is not provided with the opening mask 103 that generates parallax is referred to as a non-parallax pixel. For example, when the effective pixel area of the first image sensor 100 is about 24 mm × 16 mm, the number of pixels reaches about 12 million.

  Note that in the case of an image sensor with good light collection efficiency and photoelectric conversion efficiency, the microlens 101 may not be provided. In the case of a back-illuminated image sensor, the wiring layer 105 is provided on the side opposite to the photoelectric conversion element 108.

  There are various variations in the combination of the color filter 102 and the aperture mask 103. In FIG. 2A, the color filter 102 and the opening mask 103 can be integrally formed if the opening 104 of the opening mask 103 has a color component. In addition, when a specific pixel is a pixel that acquires luminance information of a subject, the corresponding color filter 102 may not be provided for the pixel. Or you may arrange | position the transparent filter which does not give coloring so that the substantially all wavelength band of visible light may be permeate | transmitted.

  When the pixel from which luminance information is acquired is a parallax pixel, that is, when the parallax image is output at least once as a monochrome image, the configuration of the image sensor 120 shown in FIG. 2B can be employed. That is, the screen filter 121 in which the color filter part 122 that functions as a color filter and the opening mask part 123 having the opening 104 are integrally formed may be disposed between the microlens 101 and the wiring layer 105. it can.

  The screen filter 121 is formed by, for example, blue-green-red coloring in the color filter portion 122 and black in the opening mask portion 123 other than the opening portion 104. Since the image sensor 120 that employs the screen filter 121 has a shorter distance from the microlens 101 to the photoelectric conversion element 108 than the first image sensor 100, the light collection efficiency of the subject light flux is high.

  Next, the relationship between the opening 104 of the opening mask 103 and the generated parallax will be described. FIG. 3 is a schematic diagram illustrating a state in which a part of the first image sensor 100 is enlarged. Here, in order to simplify the explanation, the color arrangement of the color filter 102 is not considered until the reference is resumed later. In the following description that does not refer to the color arrangement of the color filter 102, it can be considered that the image sensor is a collection of only parallax pixels having the color filter 102 of the same color (including the case of being transparent). Therefore, the repetitive pattern described below may be considered as an adjacent pixel in the color filter 102 of the same color.

  As shown in FIG. 3, the opening 104 of the opening mask 103 is provided so as to be relatively shifted with respect to each pixel. In the adjacent pixels, the openings 104 are provided at positions displaced from each other.

  In the example shown in the drawing, six types of opening masks 103 that are shifted in the left-right direction are prepared as the positions of the openings 104 for the respective pixels. The entire first image sensor 100 has a two-dimensional and periodic arrangement of photoelectric conversion element groups each including a set of six parallax pixels each having an aperture mask 103 that gradually shifts from the left side to the right side of the drawing. ing. That is, it can be said that the first image sensor 100 is configured by periodically laying a repeating pattern 110 including a set of photoelectric conversion element groups.

  FIG. 4 is a conceptual diagram illustrating the relationship between the parallax pixels and the subject. In particular, FIG. 4A shows a photoelectric conversion element group of a repetitive pattern 110t arranged at the center orthogonal to the photographing optical axis 21 in the first image pickup element 100, and FIG. 4B is arranged in the peripheral portion. The photoelectric conversion element group of the repeating pattern 110u which has shown is shown typically. The subject 30 in FIGS. 4A and 4B is in the in-focus position with respect to the photographic lens 20. FIG. 4C schematically shows a relationship when the subject 31 existing at the out-of-focus position with respect to the photographing lens 20 is captured corresponding to FIG.

  First, the relationship between the parallax pixels and the subject when the photographing lens 20 captures the subject 30 that is in focus will be described. The subject luminous flux passes through the pupil of the photographing lens 20 and is guided to the first image sensor 100. Six partial areas Pa to Pf are defined for the entire cross-sectional area through which the subject luminous flux passes. For example, in the pixel at the left end of the sheet of the photoelectric conversion element group constituting the repetitive patterns 110t and 110u, only the subject luminous flux emitted from the partial region Pf reaches the photoelectric conversion element 108 as can be seen from the enlarged view. The position of the opening 104f of the opening mask 103 is determined. Similarly, the position of the opening 104e corresponding to the partial area Pe, the position of the opening 104d corresponding to the partial area Pd, and the position of the opening 104c corresponding to the partial area Pc toward the pixel at the right end of the page. The position of the opening 104b is determined corresponding to the partial area Pb, and the position of the opening 104a is determined corresponding to the partial area Pa.

  In other words, the position of the opening 104f is determined by the inclination of the principal ray Rf of the subject light beam emitted from the partial region Pf, which is defined by the relative positional relationship between the partial region Pf and the leftmost pixel on the paper surface. It may be said that there is. Then, when the photoelectric conversion element 108 receives the subject luminous flux from the subject 30 existing at the in-focus position via the opening 104f, the subject luminous flux is coupled on the photoelectric conversion element 108 as shown by the dotted line. Image. Similarly, toward the pixel at the right end of the page, the position of the opening 104e is determined by the inclination of the principal ray Re, the position of the opening 104d is determined by the inclination of the principal ray Rd, and the position of the opening 104c is determined by the inclination of the principal ray Rc. It can be said that the position of the opening 104b is determined by the inclination of the principal ray Rb, and the position of the opening 104a is determined by the inclination of the principal ray Ra.

  As shown in FIG. 4A, the light beam emitted from the minute region Ot on the subject 30 that intersects the optical axis 21 among the subject 30 existing at the in-focus position passes through the pupil of the photographing lens 20. Then, each pixel of the photoelectric conversion element group constituting the repetitive pattern 110t is reached. That is, each pixel of the photoelectric conversion element group constituting the repetitive pattern 110t receives a light beam emitted from one minute region Ot through each of the six partial regions Pa to Pf. Although the minute region Ot has an extent corresponding to the positional deviation of each pixel of the photoelectric conversion element group constituting the repetitive pattern 110t, it can be approximated to substantially the same object point. Similarly, as shown in FIG. 4B, the light beam emitted from the minute region Ou on the subject 30 that is separated from the optical axis 21 among the subject 30 that exists at the in-focus position passes through the pupil of the photographing lens 20. It passes through and reaches each pixel of the photoelectric conversion element group constituting the repetitive pattern 110u. That is, each pixel of the photoelectric conversion element group constituting the repetitive pattern 110u receives a light beam emitted from one minute region Ou through each of the six partial regions Pa to Pf. Similarly to the micro area Ot, the micro area Ou has an extent corresponding to the positional deviation of each pixel of the photoelectric conversion element group constituting the repetitive pattern 110u, but substantially the same object point. Can be approximated.

  That is, as long as the subject 30 exists at the in-focus position, the microregion captured by the photoelectric conversion element group differs depending on the position of the repetitive pattern 110 on the first imaging element 100, and constitutes the photoelectric conversion element group. Each pixel captures the same minute area through different partial areas. In each repetitive pattern 110, corresponding pixels receive the subject luminous flux from the same partial area. In other words, in the drawing, for example, the pixel at the left end of each of the repetitive patterns 110t and 110u receives the subject luminous flux from the same partial region Pf.

  In the repetitive pattern 110t arranged in the center orthogonal to the photographing optical axis 21, the position of the opening 104f where the leftmost pixel on the paper surface receives the subject light beam from the partial region Pf and the repetitive pattern 110u arranged in the peripheral portion on the paper surface. Strictly speaking, the position of the opening 104f where the left end pixel receives the subject light flux from the partial region Pf is different. However, from a functional point of view, these can be treated as the same type of aperture mask in terms of an aperture mask for receiving the subject light flux from the partial region Pf. Therefore, in the example of FIG. 4, it can be said that each of the parallax pixels arranged on the first image sensor 100 includes one of six types of aperture masks.

  Next, the relationship between the parallax pixels and the subject when the photographing lens 20 captures the subject 31 existing in the out-of-focus state will be described. Also in this case, the subject luminous flux from the subject 31 present at the out-of-focus position passes through the six partial areas Pa to Pf of the pupil of the photographing lens 20 and reaches the first image sensor 100. However, the subject light flux from the subject 31 existing at the out-of-focus position forms an image at another position, not on the photoelectric conversion element 108. For example, as shown in FIG. 4C, when the subject 31 is located farther from the first image sensor 100 than the subject 30, the subject luminous flux forms an image on the subject 31 side with respect to the photoelectric conversion element 108. . Conversely, when the subject 31 is located closer to the first image sensor 100 than the subject 30, the subject luminous flux forms an image on the opposite side of the subject 31 from the photoelectric conversion element 108.

  Therefore, the subject luminous flux emitted from the minute region Ot ′ among the subjects 31 existing at the out-of-focus position depends on which of the six partial regions Pa to Pf, the corresponding pixels in the different sets of repetitive patterns 110. To reach. For example, as shown in the enlarged view of FIG. 4C, the subject luminous flux that has passed through the partial region Pd is incident on the photoelectric conversion element 108 having the opening 104d included in the repeated pattern 110t ′ as the principal ray Rd ′. To do. Even if the subject light beam is emitted from the minute region Ot ′, the subject light beam that has passed through another partial region does not enter the photoelectric conversion element 108 included in the repetitive pattern 110t ′, and the repetitive pattern in the other repetitive pattern. The light enters the photoelectric conversion element 108 having a corresponding opening. In other words, the subject luminous flux reaching each photoelectric conversion element 108 constituting the repetitive pattern 110t ′ is a subject luminous flux radiated from different minute areas of the subject 31. That is, a subject luminous flux having a principal ray as Rd ′ is incident on 108 corresponding to the opening 104d, and the principal rays are Ra +, Rb +, Rc +, Re +, Rf + to the photoelectric conversion elements 108 corresponding to the other openings. The subject luminous flux is incident, and these subject luminous fluxes are subject luminous fluxes radiated from different minute regions of the subject 31. Such a relationship is the same in the repeated pattern 110u arranged in the peripheral portion in FIG.

  Then, when viewed as a whole of the first image sensor 100, for example, the subject image A captured by the photoelectric conversion element 108 corresponding to the opening 104a and the subject image D captured by the photoelectric conversion element 108 corresponding to the opening 104d. If there is an image with respect to the subject existing at the in-focus position, there will be no shift, and if there is an image with respect to the subject existing in the out-of-focus position, there will be a shift. Then, the direction and amount of the shift are determined by how much the subject existing at the out-of-focus position is shifted from the focus position and by the distance between the partial area Pa and the partial area Pd. That is, the subject image A and the subject image D are parallax images. Since this relationship is the same for the other openings, six parallax images are formed corresponding to the openings 104a to 104f.

  Therefore, when the outputs of the pixels corresponding to each other in each of the repetitive patterns 110 configured in this way are collected, a parallax image is obtained. That is, the output of the pixel that has received the subject light beam emitted from a specific partial area among the six partial areas Pa to Pf forms a parallax image.

  FIG. 5 is a conceptual diagram illustrating processing for generating a parallax image. The figure shows, in order from the left column, the generation of the parallax image data Im_f generated by collecting the outputs of the parallax pixels corresponding to the opening 104f, the generation of the parallax image data Im_e by the output of the opening 104e, the opening State of generation of parallax image data Im_d by output of 104d, state of generation of parallax image data Im_c by output of opening 104c, state of generation of parallax image data Im_b by output of opening 104b, parallax by output of opening 104a This represents how the image data Im_a is generated. First, how the parallax image data Im_f is generated by the output of the opening 104f will be described.

  The repetitive pattern 110 composed of a photoelectric conversion element group including a set of six parallax pixels is arranged in a horizontal row on the paper parallel to the x-axis direction. Accordingly, the parallax pixels having the opening 104f exist every six pixels in the x-axis direction and continuously in the y-axis direction on the first image sensor 100. Each of these pixels receives the subject luminous flux from different microregions as described above. Therefore, when the outputs of these parallax pixels are collected and arranged, an x-axis direction, that is, a horizontal parallax image is obtained.

  However, since each pixel of the first image sensor 100 in the present embodiment is a square pixel, simply gathering results in the number of pixels in the x-axis direction being thinned out to 1/6, and in the y-axis direction. Vertical image data is generated. Therefore, by performing an interpolation process so that the number of pixels is 6 times in the x-axis direction, the parallax image data Im_f is generated as an image with an original aspect ratio. However, since the parallax image data before interpolation processing is an image that is thinned out to 1/6 in the x-axis direction, the horizontal resolution in the x-axis direction is lower than the vertical resolution in the y-axis direction. That is, it can be said that the number of parallax image data to be generated and the resolution are in a reciprocal relationship.

  Similarly, parallax image data Im_e to parallax image data Im_a are obtained. That is, the digital camera 10 can generate a 6-view horizontal parallax image having parallax in the x-axis direction.

  In the above example, an example in which a horizontal row parallel to the x-axis direction is periodically arranged as a repeating pattern 110 has been described, but the repeating pattern 110 is not limited to this. FIG. 6 is a diagram illustrating another example of the repeated pattern 110.

  FIG. 6A shows an example in which the repeating pattern 110 includes 6 pixels in the y-axis direction. However, the positions of the respective openings 104 are determined so as to gradually shift from the left side to the right side of the drawing from the parallax pixel at the top of the drawing to the bottom. A 6-view horizontal parallax image that gives parallax in the x-axis direction can also be generated by the repeated pattern 110 arranged in this way. In this case, compared to the repetitive pattern 110 in FIG. 3, it can be said that the repetitive pattern maintains the horizontal resolution in the x-axis direction instead of sacrificing the vertical resolution in the y-axis direction.

  FIG. 6B shows an example in which the 6 patterns adjacent to each other in the diagonal direction on the paper are used as the repeated pattern 110. The positions of the openings 104 are determined so as to gradually shift from the left side to the right side of the drawing from the parallax pixel at the upper left corner of the drawing toward the lower right. A 6-view horizontal parallax image that gives parallax in the x-axis direction can also be generated by the repeated pattern 110 arranged in this way. In this case, compared to the repetitive pattern 110 in FIG. 3, it can be said that the repetitive pattern increases the number of horizontal parallax images while maintaining the vertical resolution in the y-axis direction and the horizontal resolution in the x-axis direction to some extent.

  Comparing the repetitive pattern 110 in FIG. 3 with the repetitive pattern 110 in FIGS. 6A and 6B, when all generate parallax images with 6 viewpoints, one image from the whole which is not a parallax image is generated. It can be said that this is the difference between sacrificing the resolution in the y-axis direction or the x-axis direction with respect to the resolution when outputting. In the case of the repetitive pattern 110 in FIG. 3, the horizontal resolution in the x-axis direction is set to 1/6. In the case of the repetitive pattern 110 in FIG. 6A, the vertical resolution in the y-axis direction is set to 1/6. 6B has a configuration in which the y-axis direction is 1/3 and the x-axis direction is 1/2. In any case, one opening 104a to 104f is provided corresponding to each pixel in one pattern, and the subject luminous flux is received from one of the corresponding partial areas Pa to Pf. It is configured as follows. Accordingly, the parallax amount is the same for any of the repeated patterns 110.

  In the above example, the case of generating a horizontal parallax image that gives a parallax in the horizontal direction has been described. Of course, a vertical parallax image that gives a parallax in the vertical direction can also be generated, and the parallax can be generated in a horizontal and vertical two-dimensional direction. A horizontal / vertical parallax image to be given can also be generated. FIG. 7 is a diagram illustrating an example of a two-dimensional repetitive pattern 110.

  According to the example of FIG. 7, the repetitive pattern 110 is formed by using 36 pixels of 6 y-axis pixels and 6 x-axis pixels as a set of photoelectric conversion element groups. As the positions of the openings 104 for the respective pixels, 36 types of opening masks 103 that are mutually shifted in the y-axis and x-axis directions are prepared. Specifically, each opening 104 gradually shifts from the upper side of the drawing to the lower side of the repetitive pattern 110 from the upper end pixel to the lower end pixel, and at the same time from the left side of the drawing toward the right end pixel. Positioned to shift gradually to the right.

  The first image sensor 100 having such a repeating pattern 110 can output a parallax image of 36 viewpoints that gives parallax in the vertical direction and the horizontal direction. Of course, the pattern 110 is not limited to the example in FIG. 7, and the repetitive pattern 110 can be determined so as to output parallax images with various viewpoints.

  In the above description, a rectangle is adopted as the shape of the opening 104. In particular, in an array that gives a parallax in the horizontal direction, the amount of light that is guided to the photoelectric conversion element 108 is secured by making the width in the y-axis direction that is not shifted wider than the width in the x-axis direction that is the shifting direction. . However, the shape of the opening 104 is not limited to a rectangle.

  FIG. 8 is a diagram illustrating another shape of the opening 104. In the figure, the shape of the opening 104 is circular. In the case of a circular shape, an unscheduled subject light beam can be prevented from entering the photoelectric conversion element 108 as stray light because of the relative relationship with the microlens 101 having a hemispherical shape.

  Next, the color filter 102 and the parallax image will be described. FIG. 9 is a diagram illustrating the Bayer arrangement. As shown in the figure, the Bayer array is an array in which a green filter is assigned to two pixels at the upper left and lower right of the paper, a red filter is assigned to one pixel at the lower left of the paper, and a blue filter is assigned to one pixel at the upper right of the paper. Here, the upper left pixel on the paper to which the green filter is assigned is the Gb pixel, and the lower right pixel on the paper to which the green filter is assigned is the Gr pixel. In addition, a pixel to which a red filter is assigned is an R pixel, and a pixel to which blue is assigned is a B pixel. The horizontal direction of the paper on which Gb pixels and B pixels are arranged is defined as Gb row, and the horizontal direction of the paper on which R pixels and Gr pixels are aligned is defined as Gr row. Further, the vertical direction of the paper on which Gb pixels and R pixels are arranged is referred to as Gb column, and the vertical direction of the paper on which B pixels and Gr pixels are arranged is referred to as Gr column.

  With respect to such an arrangement of the color filters 102, an enormous number of repetitive patterns 110 can be set depending on what color pixels the parallax pixels and non-parallax pixels are allocated to. If the outputs of pixels without parallax are collected, photographic image data having no parallax can be generated in the same way as normal photographic images. Therefore, if the ratio of pixels without parallax is relatively increased, a 2D image with high resolution can be output. In this case, since the number of parallax pixels is relatively small, stereoscopic information is reduced as a 3D image including a plurality of parallax images. Conversely, if the ratio of parallax pixels is increased, stereoscopic information increases as a 3D image, but non-parallax pixels decrease relatively, so a 2D image with low resolution is output.

  In such a trade-off relationship, a repetitive pattern 110 having various characteristics is set depending on which pixel is a parallax pixel or a non-parallax pixel.

  FIG. 10 is a diagram illustrating a parallax image capturing process including the first image sensor 100 and the second image sensor 300. In the example shown in FIG. 10, the spectroscopic unit 50 is configured by a half mirror. As shown in FIG. 10, in the first image sensor 100, the upper left Gb pixel and the lower right Gr pixel are configured as parallax Lt pixels, and the lower left R pixel and the upper right B pixel are It is configured as a pixel without parallax. The first image sensor 100 has the same configuration as the second image sensor 300 rotated by 180 ° around the normal line. Specifically, in the second image sensor 300, the upper left Gr pixel and the lower right Gb pixel are configured as parallax Rt pixels, and the upper right R pixel and the lower left B pixel are non-parallax pixels. It is configured. Thereby, since the same image sensor can be applied as the 1st image sensor 100 and the 2nd image sensor 300, a manufacturing process and manufacturing equipment can be simplified. 10 is an example in which the wavelength characteristics of the color filter 102 and the position of the opening 104 of the opening mask 103 are different from each other at corresponding positions in the array. The first image sensor 100 and the second image sensor 300 are provided with an aperture mask and a color filter.

  A parallax image imaging process by the digital camera 10 having the first imaging element 100 and the second imaging element 300 will be described. First, the first image sensor 100 captures an image of a subject and outputs an electrical signal using the first spectrum DL1 split by the spectroscopic unit 50, and the second image sensor 300 is split by the spectroscopic unit 50. The spectral DL2 captures an image of the subject and outputs an electrical signal. These electric signals are converted into image data by the A / D conversion circuits 202 and 302, respectively, and stored in the memories 203 and 303. The image extraction unit 231 of the image processing unit 205 acquires each digitized image data.

  The image extraction unit 231 of the image processing unit 205 executes extraction processing. In the extraction process, the image extraction unit 231 includes the image data RN1 output from the R pixel of the first image sensor 100, the image data GLt1 output from the Gb pixel and the Gr pixel, and the image data BN1 output from the B pixel. And the image data GLt2 output from the Gb pixel and the Gr pixel are separated from the image data of the first image sensor 100 stored in the memory 203 and extracted. Image data RN1, image data GLt1, and image data BN1 are color data. The image data GLt2 is parallax data. The image extraction unit 231 also includes image data RN2 output from the R pixel of the second image sensor 300, image data GRt1 output from the Gb pixel and Gr pixel, image data BN2 output from the B pixel, The image data GRt2 output from the Gb pixel and the Gr pixel is separated from the image data of the second image sensor 300 stored in the memory 303 and extracted. Image data RN2, image data GRt1, and image data BN2 are color data. The image data GRt2 is parallax data. Each image data includes an empty grid which is a pixel without pixel values of color data and parallax data.

  Next, the image complementing unit 232 of the image processing unit 205 executes complementing processing. In the complementing process, the image complementing unit 232 synthesizes the RN1 image data RN1 of the first image sensor 100 and the image data RN2 of the second image sensor 300 to create R composite data including R color data of the R image. . Furthermore, the image complementing unit 232 generates pixel values of the R color data of the empty lattice based on the R color data of a plurality of pixels adjacent to the empty lattice in the R composite data of the R image. The image complementing unit 232 generates plain image data RN in which all pixels have R color data by generating and complementing the R color data of each empty lattice. Similarly, the image complementing unit 232 generates plain image data GN in which all pixels have G color data from the image data GLt1 and the image data GRt1. The image complementing unit 232 generates plain image data BN in which all pixels have B color data from the image data BN1 and the image data BN2.

  Further, in the complementing process, the image complementing unit 232 generates plain image data Lt in which all the pixels have Lt parallax data by complementing the empty lattice of the image data GLt2 of the first image sensor 100. Similarly, the image complementing unit 232 generates plane image data Rt having Rt parallax data by complementing the empty lattice of the image data GRt2 of the second image sensor 300.

  Next, the image composition unit 233 of the image processing unit 205 executes composition processing. In the combining process, the image combining unit 233 combines the plane image data RN and the plane image data Lt to generate image data RLt in which all pixels have R color data and L parallax data. Similarly, the image combining unit 233 combines the plane image data RN and the plane image data Rt to generate image data RRt in which all pixels have R color data and R parallax data. The image compositing unit 233 generates image data Gt and image data GRt in which all pixels have G color data and L parallax data or R parallax data from the plane image data GN and the plane image data Lt or the plane image data Rt. Generate. The image composition unit 233 generates image data BLt and image data BRt in which all pixels have B color data and L parallax data or R parallax data from the plain image data BN and the plain image data Lt or the plain image data Rt. Generate. Thereby, the image composition unit 233 generates image data in which all pixels have RGB color data and RL parallax data.

  FIG. 11 is a diagram illustrating a parallax image imaging process including another first imaging element 100 and a second imaging element 300. In the example shown in FIG. 11, the spectroscopic unit 50 includes a spectroscopic mirror such as a dichroic mirror and a dichroic prism. As shown in FIG. 11, in the first image sensor 100, all of the upper left Gb pixel, the lower left R pixel, the upper right B pixel, and the lower right Gr pixel are configured as non-parallax pixels. In the second image sensor 300, all the pixels are configured as IR pixels capable of receiving infrared rays. In the second image sensor 300, the upper left IR pixel and the lower right IR pixel are configured as parallax Lt pixels, and the lower left IR pixel and the upper right IR pixel are configured as parallax Rt pixels. The spectroscopic unit 50 separates the visible light included in the subject light beam as the first spectroscopic DL1 that travels to the first image sensor 100 and the infrared light as the second spectroscopic DL2 that travels to the second image sensor 300. Thereby, since the 2nd image sensor 300 can receive the infrared light which the 1st image sensor 100 does not need, and can produce | generate parallax data, the reduction of the light quantity of the visible light which the 1st image sensor 100 receives can be suppressed. . Since the second image sensor 300 can receive infrared light and generate parallax data, all infrared light included in the subject light beam can be split into the second image sensor 300. As a result, the first image sensor 100 is less susceptible to the influence of infrared light that becomes noise. Thereby, the quality of the image data output from the first image sensor 100 can be improved. In the second image sensor 300, a plurality of types of aperture masks are periodically arranged.

  In the digital camera 10, after the first image sensor 100 and the second image sensor 300 image a subject, the image extraction unit 231 of the image processing unit 205 executes an extraction process. In the extraction process, the image extraction unit 231 selects image data RN1 having R color data, image data GN1 having G color data, and image data BN1 having B color data from the image data output from the first image sensor 100. Is extracted and generated. The image data RN1, the image data GN1, and the image data BN1 include vacancies.

  The image extraction unit 231 extracts image data IRLt1 including Lt parallax data and image data IRRt1 including Rt parallax data from the image data output from the second imaging element 300. The image data IRLt1 and the image data IRRt1 include vacancies.

  Next, the image complementing unit 232 performs a complementing process. In the complementing process, the image complementing unit 232 supplements the color data of each empty grid of the image data RN1, the image data GN1, and the image data BN1 with the color data of the pixels adjacent to each empty grid. Thereby, the image complementing unit 232 has plain image data RN in which all pixels have R color data, plain image data GN in which all pixels have G color data, and plain image data BN in which all pixels have B color data. Is generated. The image complementing unit 232 supplements the parallax data of the image data IRLt1 and the respective empty lattices of the image data IRRt1 with the disparity data of the pixels adjacent to each empty lattice. Accordingly, the image complementing unit 232 generates plane image data IRLt in which all pixels have L parallax data, and plane image data IRRt in which all pixels have R parallax data.

  Next, the image composition unit 233 executes a composition process. In the combining process, the image combining unit 233 includes image data RLt in which all pixels have R color data and L parallax data or R parallax data from the plane image data RN and the plane image data Lt or the plane image data Rt. Image data RRt is generated. The image compositing unit 233 generates image data Gt and image data GRt in which all pixels have G color data and L parallax data or R parallax data from the plane image data GN and the plane image data Lt or the plane image data Rt. Generate. The image composition unit 233 generates image data BLt and image data BRt in which all pixels have B color data and L parallax data or R parallax data from the plain image data BN and the plain image data Lt or the plain image data Rt. Generate. Thereby, the image composition unit 233 generates image data in which all pixels have RGB color data and RtLt parallax data.

  FIG. 12 is a diagram illustrating another first image sensor 100 and second image sensor 300. In the example shown in FIG. 12, the spectroscopic unit 50 is configured by a half mirror. The first image sensor 100 and the second image sensor 300 shown in FIG. 12 are in a mirror image relationship with each other. The first image sensor 100 shown in FIG. 12 has the same configuration as the first image sensor 100 shown in FIG. In the second image sensor 300 shown in FIG. 12, the upper right Gb pixel and the lower left Gr pixel are configured as parallax Rt pixels, and the upper left B pixel and the lower right R pixel are configured as non-parallax pixels. ing.

  FIG. 13 is a partial configuration diagram of the digital camera imaging optical unit 610 in which the spectroscopic unit 650 splits the subject light flux into three. As shown in FIG. 13, the spectroscopic unit 650 splits the subject light beam into the first spectroscopic DL1, the second spectroscopic DL2, and the third spectroscopic DL3. An example of the spectroscopic unit 650 is a dichroic prism. The spectroscopic unit 650 includes a first spectroscopic member 652, a second spectroscopic member 654, and a third spectroscopic member 656. The third imaging element 400, the second imaging element 300, and the first imaging element 100 are arranged so as to face each end face of the first spectral member 652, the second spectral member 654, and the third spectral member 656. The third image sensor 400 has the same configuration as the first image sensor 100 or the second image sensor 300 described above. Note that the configurations of the first image sensor 100, the second image sensor 300, and the third image sensor 400 may be switched as appropriate.

  One surface of the first light separating member 652 faces the photographing lens 20. The other surface of the first beam splitting member 652 faces one surface of the second beam splitting member 654. The other surface of the second light separating member 654 faces one surface of the third light separating member 656. After the subject luminous flux enters the first light splitting member 652, a part of the green light flux, for example, is reflected by the other surface of the first light splitting member 652, and the rest is transmitted. The reflected subject luminous flux is reflected on one surface of the first light splitting member 652 and proceeds to the third image sensor 400 as the third light split DL3. Of the subject light flux that has passed through one surface of the first light splitting member 652, for example, the remainder of the green light flux is reflected by the other surface of the second light splitting member 654 and then reflected by one surface of the second light splitting member 654. Then, the remainder of the green light flux travels to the second image sensor 300 as the second spectral DL2. The remaining blue and red light fluxes of the subject light flux that has passed through one surface of the first light splitting member 652 pass through the other face of the second light splitting member 654 and reach the first image sensor 100 as the first light split DL1.

  FIG. 14 is a front view illustrating the first image sensor 100, the second image sensor 300, and the third image sensor 400 in the digital camera imaging optical unit 610 of FIG. As shown in FIG. 14, in the first image sensor 100, all the pixels are Gb pixels and are configured as parallax Lt pixels. In the second image sensor 300, all the pixels are Gr pixels and are configured as parallax Rt pixels. In the third image sensor 400, the upper left pixel and the lower right pixel in the drawing are B pixels and are configured as non-parallax pixels. In the third image sensor 400, the lower left pixel and the upper right pixel in the drawing are R pixels and are configured as non-parallax pixels.

  FIG. 15 is a front view illustrating another first image sensor 100, second image sensor 300, and third image sensor 400 in the digital camera imaging optical unit 610 of FIG. As shown in FIG. 15, in the first image sensor 100, all the pixels are G pixels and are configured as non-parallax pixels. In the second image sensor 300, the upper left pixel and the lower right pixel in the drawing are B pixels and are configured as non-parallax pixels. In the second image sensor 300, the lower left pixel and the upper right pixel in the drawing are R pixels and are configured as non-parallax pixels. In the third image sensor 400, the upper left pixel and the lower right pixel in the drawing are IR pixels and are configured as parallax Lt pixels. In the third image sensor 400, the lower left pixel and the upper right pixel in the drawing are IR pixels and are configured as parallax Rt pixels. In the example illustrated in FIG. 15, the first spectral DL1 and the second spectral DL2 are visible light, and the third spectral DL3 is infrared light.

  FIG. 16 is a partial configuration diagram of the digital camera imaging optical unit 710 in which the spectroscopic unit 750 splits the subject light flux into four. As shown in FIG. 16, the spectroscopic unit 650 splits the subject light beam into a first light beam DL1, a second light beam DL2, a third light beam DL3, and a fourth light beam DL4. An example of the spectroscopic unit 750 is a dichroic prism. The spectroscopic unit 750 includes a first spectroscopic member 752, a second spectroscopic member 754, a third spectroscopic member 756, and a fourth spectroscopic member 758. An example of the first beam splitting member 752, the second beam splitting member 754, the third beam splitting member 756, and the fourth beam splitting member 758 is a prism. The fourth imaging element 500, the third imaging element 400, and the second imaging element so as to face each end face of the first spectral member 752, the second spectral member 754, the third spectral member 756, and the fourth spectral member 758. 300 and the 1st image pick-up element 100 are arrange | positioned. The fourth image sensor 500 has the same configuration as the first image sensor 100 or the second image sensor 300 described above. Note that the configurations of the first image sensor 100, the second image sensor 300, the third image sensor 400, and the fourth image sensor 500 may be switched as appropriate.

  One surface of the first light separating member 752 faces the photographing lens 20. The other surface of the first beam splitting member 752 faces one surface of the second beam splitting member 754. The other surface of the second light separating member 754 faces one surface of the third light separating member 756. The other surface of the third light separating member 756 faces one surface of the fourth light separating member 758.

  After the subject luminous flux is incident on the first beam splitting member 752, a part of the subject beam is reflected by the other surface of the first beam splitting member 752, and the rest is transmitted. The reflected subject luminous flux is reflected by one surface of the first light separating member 752 and guided to the fourth image sensor 500. Part of the transmitted subject luminous flux is reflected on the other surface of the second light-splitting member 754 or the other surface of the third light-splitting member 756 and then guided to the third image sensor 400 or the second image sensor 300. The subject light flux that has passed through the first light splitting member 752, the second light splitting member 754, and the third light splitting member 756 is guided to the first image sensor 100.

  FIG. 17 is a front view illustrating the first image sensor 100, the second image sensor 300, the third image sensor 400, and the fourth image sensor 500 in the digital camera imaging optical unit 710 of FIG. As shown in FIG. 17, in the first image sensor 100, all the pixels are G pixels and are configured as non-parallax pixels. In the second image sensor 300, all the pixels are R pixels and are configured as non-parallax pixels. In the third image sensor 400, all the pixels are B pixels and are configured as non-parallax pixels. In the fourth image sensor 500, the upper left pixel and the lower right pixel in the drawing are IR pixels, which are configured as parallax Lt pixels. In the fourth image sensor 500, the lower left pixel and the upper right pixel in the drawing are IR pixels, which are configured as parallax Rt pixels. In the example shown in FIG. 17, the first spectrum DL1, the second spectrum DL2, and the third spectrum DL3 are visible light, and the fourth spectrum DL4 is infrared light.

  FIG. 18 is a front view illustrating another first image sensor 100, second image sensor 300, third image sensor 400, and fourth image sensor 500 in the digital camera imaging optical unit 710 of FIG. As shown in FIG. 18, in the first image sensor 100, all the pixels are G pixels and are configured as parallax Lt pixels. In the second image sensor 300, all the pixels are G pixels and are configured as parallax Rt pixels. In the third image sensor 400, the upper left pixel and the lower right pixel in the drawing are B pixels and are configured as non-parallax pixels. In the third image sensor 400, the lower left pixel and the upper right pixel in the drawing are R pixels and are configured as non-parallax pixels. The fourth image sensor 500 has the same configuration as the fourth image sensor 500 in FIG. The first image sensor 100 and the second image sensor 300 shown in FIG. 18 have the same wavelength characteristics of the color filter 102 at the corresponding positions in the array, and the positions of the openings 104 of the opening mask 103 are different from each other. It is an example. In the example illustrated in FIG. 18, the first spectrum DL1, the second spectrum DL2, and the third spectrum DL3 are visible light, and the fourth spectrum DL4 is infrared light.

  FIG. 19 is a front view illustrating another first image sensor 100, second image sensor 300, third image sensor 400, and fourth image sensor 500 in the digital camera imaging optical unit 710 of FIG. The first image sensor 100 and the second image sensor 300 shown in FIG. 19 have the same configuration as the first image sensor 100 and the second image sensor 300 shown in FIG. In the third image sensor 400, the upper left pixel and the lower right pixel in the drawing are R pixels and are configured as parallax Lt pixels. In the third image sensor 400, the lower left pixel and the upper right pixel in the drawing are R pixels, and are configured as parallax Rt pixels. In the fourth image sensor 500, the upper left pixel and the lower right pixel in the drawing are B pixels, which are configured as parallax Lt pixels. In the fourth image sensor 500, the lower left pixel and the upper right pixel in the drawing are B pixels, which are configured as parallax Rt pixels. Note that the third image sensor 400 and the fourth image sensor 500 shown in FIG. 19 have the same position of the aperture 104 of the aperture mask 103 at the corresponding positions in the array, and the wavelength characteristics of the color filter 102 are different from each other. It is an example.

  FIG. 20 is a front view illustrating another first image sensor 100, second image sensor 300, third image sensor 400, and fourth image sensor 500 in the digital camera imaging optical unit 710 of FIG. The first image sensor 100 and the second image sensor 300 shown in FIG. 20 have the same configuration as the first image sensor 100 and the second image sensor 300 shown in FIG. In the third image sensor 400, all the pixels are R pixels and are configured as non-parallax pixels. In the fourth image sensor 500, all the pixels are B pixels, and are configured as non-parallax pixels.

  You may change the magnitude | size of the opening of a parallax pixel in each image pick-up element. FIG. 21 is a front view for explaining an image sensor in which the size of the opening is changed. As shown in FIG. 21, the upper left pixel and the lower right pixel of the first image sensor 100 are the Gb pixel and the Gr pixel, and are the parallax Lt pixels. The upper right pixel and lower left pixel of the first image sensor 100 are the B pixel and the R pixel, and are the parallax Rt pixels. On the other hand, the second image sensor 300 has the same arrangement of color and parallax as the first image sensor 100. However, the area of the opening of each pixel of the second image sensor 300 is smaller than the area of the opening of each pixel of the first image sensor 100. For example, the area of the opening of each pixel of the second image sensor 300 is half of the area of the opening of each pixel of the first image sensor 100. Accordingly, the first image sensor 100 and the second image sensor 300 can simultaneously capture parallax images with different apertures.

  FIG. 22 is a front view for explaining another image sensor in which the size of the opening is changed. As illustrated in FIG. 22, the upper left pixel and the lower right pixel of the first image sensor 100 are the Gb pixel and the Gr pixel, and the parallax Lt pixel and the parallax Rt pixel. The upper right pixel and the lower left pixel of the first image sensor 100 are the B pixel and the R pixel, and are pixels without parallax. The area of the opening of the non-parallax pixel is, for example, twice as large as the area of the opening of the parallax Lt pixel and the parallax Rt pixel. On the other hand, the second image sensor 300 has the same arrangement of color and parallax as the first image sensor 100. However, the area of the opening of each pixel of the second image sensor 300 is smaller than the area of the opening of each pixel of the first image sensor 100. For example, the area of the opening of each pixel of the second image sensor 300 is half of the area of the opening of each pixel of the first image sensor 100. Accordingly, the first image sensor 100 and the second image sensor 300 can simultaneously capture parallax images with different apertures.

  Although various variations have been described regarding the relationship between the parallax pixels and the color filters, other variations may be used. Each variation is that at least a part of a set of photoelectric conversion elements of an image sensor on which one spectrum from the same object is incident and a photoelectric conversion element of an image sensor on which another spectrum is incident, whether or not there is a color filter, It is only necessary that at least one of the wavelength characteristics of the color filter, the presence / absence of the opening, and the position of the opening be different. Hereinafter, variations will be described with reference to the table.

  FIG. 23 is a table for explaining the types of imaging elements. As shown in the table of FIG. 23, there are four types of image sensors, element 1 to element 4. An element having a parallax pixel of “◯” has a parallax pixel, and the other elements have no parallax pixel. An element having a color pixel of “◯” has a color filter, and other elements have no color filter. An example of the element 4 is an element that has no parallax pixel and captures a monochrome image.

  FIG. 24 is a table illustrating the types of spectroscopic units. As shown in the table of FIG. 24, there are three types of spectroscopic sections, that is, spectroscopic a to spectroscopic c. Spectroscopy a separates light of all wavelengths into a plurality of light quantities regardless of the wavelength. Spectroscopy b separates light for each specific wavelength band. Spectroscopy c separates infrared rays from light in other wavelength bands.

  FIG. 25 is a table illustrating combinations of combinations when two image sensors are provided. For example, the variation Va1 is a combination of the element 1, the spectrum a, and any one of the elements 1 to 4.

  FIG. 26 is a table for explaining a variation of combinations in the case of providing three image sensors. For example, the variation Va7 is a combination of the element 1, the spectrum a, the element 1 to the element 4, the spectrum a to the spectrum c, and the element 1 to the element 4.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above-described embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

  The order of execution of each process such as operations, procedures, steps, and stages in the apparatus, system, program, and method shown in the claims, the description, and the drawings is particularly “before” or “prior to”. It should be noted that the output can be realized in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the description, and the drawings, even if it is described using “first”, “next”, etc. for convenience, it means that it is essential to carry out in this order. It is not a thing.

DESCRIPTION OF SYMBOLS 10 Digital camera 20 Shooting lens 21 Optical axis 30 Subject 31 Subject 50 Spectroscopic part 100 1st imaging device 101 Micro lens 102 Color filter 103 Aperture mask 104 Aperture 105 Wiring layer 106 Wiring 107 Aperture 108 Photoelectric conversion element 109 Substrate 110 Pattern 120 Image sensor 121 Screen filter 122 Color filter unit 123 Aperture mask unit 201 Control unit 202 A / D conversion circuit 203 Memory 204 Drive unit 205 Image processing unit 207 Memory card IF
208 Operation Unit 209 Display Unit 210 LCD Drive Circuit 211 AF Sensor 220 Memory Card 231 Image Extraction Unit 232 Image Completion Unit 233 Image Composition Unit 238 Storage Control Unit 300 Second Image Sensor 302 A / D Conversion Circuit 303 Memory 304 Drive Unit 400 3 imaging device 500 4th imaging device 610 digital camera imaging optical unit 650 spectral unit 652 first spectral member 654 second spectral member 656 third spectral member 710 digital camera imaging optical unit 750 spectral unit 752 first spectral member 754 second spectral component Member 756 Third spectral member 758 Fourth spectral member

Claims (16)

A spectroscopic unit that splits incident light from one optical system into a first spectroscopic and a second spectroscopic;
It has a first photoelectric conversion element group in which a plurality of photoelectric conversion elements that receive the first spectrum and photoelectrically convert it into a first electric signal are two-dimensionally arranged, and at least a part of the plurality of photoelectric conversion elements On the other hand, a first image pickup device provided with at least one of a color filter and an opening provided at a position through which a light beam from a specific partial region in a cross-sectional region of the incident light passes,
It has a second photoelectric conversion element group in which a plurality of photoelectric conversion elements that receive the second spectrum and photoelectrically convert it into a second electric signal are two-dimensionally arranged, and at least a part of the plurality of photoelectric conversion elements In contrast, a color filter, and a second imaging element provided with at least one of an opening provided at a position through which a light beam from a specific partial region in a cross-sectional region of the incident light passes,
At least a part of a set of the photoelectric conversion element of the first imaging element on which the first spectrum from the same object point is incident and the photoelectric conversion element of the second imaging element on which the second spectrum is incident are mutually connected to the color filter. An imaging device in which at least one of presence / absence, wavelength characteristics of the color filter, presence / absence of the opening, and position of the opening is different.   The imaging device according to claim 1, wherein the spectroscopic unit transmits part of incident light as the first spectroscopic light and reflects the rest as the second spectroscopic light.   The imaging device according to claim 2, wherein the spectroscopic unit splits the first light split and the second light split at a predetermined light amount ratio.   The imaging apparatus according to claim 1, wherein the first imaging element has the same configuration as a configuration in which the second imaging element is rotated 180 ° around a normal line.   The imaging apparatus according to claim 1, wherein the first imaging element and the second imaging element are in a mirror image relationship with each other.   The imaging device according to claim 1, wherein a color filter and an aperture mask are provided in each of the first imaging element and the second imaging element.   The said spectroscopic part permeate | transmits the light of 1st wavelength band as 1st spectrum among incident light, and reflects the light of 2nd wavelength band different from the said 1st wavelength band as said 2nd spectrum. 4. The imaging device according to any one of 3. The first spectrum is visible light,
The imaging apparatus according to claim 7, wherein the second spectrum is infrared light. The spectroscopic unit further splits the incident light into a third spectroscopic light,
It has a third photoelectric conversion element group in which a plurality of photoelectric conversion elements that receive the third spectrum and photoelectrically convert it into a third electrical signal are two-dimensionally arranged, and at least one of the photoelectric conversion elements The image pickup apparatus according to claim 1, further comprising a third image pickup element provided with at least one of a color filter and an opening provided at a position through which a light beam from a specific partial region in a cross-sectional region of incident light passes. The first spectrum and the third spectrum are visible light,
The imaging apparatus according to claim 9, wherein the second spectrum is infrared light. The spectroscopic unit further splits the incident light into a third spectroscopic and a fourth spectroscopic,
It has a third photoelectric conversion element group in which a plurality of photoelectric conversion elements that receive the third spectrum and photoelectrically convert it into a third electrical signal are two-dimensionally arranged, and at least one of the photoelectric conversion elements A third imaging element provided with at least one of a color filter and an opening provided at a position where a light beam from a specific partial region in a cross-sectional region of incident light passes;
It has a fourth photoelectric conversion element group in which a plurality of photoelectric conversion elements that receive the fourth spectrum and photoelectrically convert it into a fourth electric signal are arranged two-dimensionally, and at least one of the photoelectric conversion elements The image pickup apparatus according to claim 1, further comprising: a color filter and a fourth image pickup element provided with at least one of openings provided at a position where light beams from a specific partial area in a cross-sectional area of incident light pass. The first spectrum, the third spectrum, and the fourth spectrum are visible light,
The imaging apparatus according to claim 11, wherein the second spectrum is infrared light.   The imaging device according to claim 1, wherein a plurality of types of openings are periodically arranged in the second imaging element.   Image extraction for extracting color data of a plurality of pixels and parallax data of a plurality of pixels from data corresponding to the first electric signal of the first image sensor and the second electric signal of the second image sensor The imaging device according to claim 1, further comprising a unit.   The imaging according to claim 14, further comprising an image complementing unit that complements the color data and the parallax data to pixels that do not have the color data and the parallax data based on the extracted color data and the parallax data. apparatus.   The imaging apparatus according to claim 15, further comprising an image composition unit that synthesizes the color data of the complemented pixel and the parallax data of the corresponding pixel.

Images