JP2012212978A - Imaging element and imaging device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve the problem in which, in a conventional camera for outputting polarization image data, ordinary color image data in which the same subject is captured cannot be output together with the polarization image data.SOLUTION: In order to solve the problem, an imaging element comprises: photoelectric conversion elements that photoelectrically convert incident light into electric signals and are two-dimensionally arranged; color filters provided to at least part of the photoelectric conversion elements in one-to-one correspondence, respectively; and photonic crystal polarizers provided to at least part of the photoelectric conversion elements in one-to-one correspondence, respectively.

Description

  The present invention relates to an imaging element and an imaging apparatus.

In the field of imaging technology using polarized light, a camera in which a patterned photonic crystal polarizer is combined on an image sensor is known. By analyzing the image data captured by this camera, it is possible to obtain the polarization distribution information of the reflected light reflected from the subject surface. This polarization distribution information is used, for example, for detecting the tilt of the subject surface.
[Prior art documents]
[Non-patent literature] Shojiro Kawakami, Takayuki Kawashima et al., "Development of polarization imaging camera using photonic crystal polarizer" 32nd Optical Symposium, Lecture No. 3, July 5, 2007

  According to the camera described above, the only image data obtained by a single photographing operation is polarized image data from which polarization information can be acquired. Therefore, color image data corresponding to the polarized image data could not be obtained. Even if a camera that outputs polarized image data and a camera that outputs normal color image data are photographed side by side, it is difficult to synchronize the photographing timing and adjust the angle of view.

  The imaging device according to the first aspect of the present invention has a one-to-one correspondence with each of a two-dimensionally arranged photoelectric conversion device that photoelectrically converts incident light into an electric signal and at least some of the photoelectric conversion devices. And a photonic crystal polarizer provided in one-to-one correspondence with each of at least some of the photoelectric conversion elements. The imaging device in the 2nd aspect of this invention is equipped with said imaging device.

  It should be noted that the above summary of the invention does not enumerate all the necessary features of the present invention. In addition, a sub-combination of these feature groups can also be an invention.

It is a figure explaining the structure of the digital camera which concerns on embodiment of this invention. It is the schematic showing the cross section of the image pick-up element which concerns on embodiment of this invention. It is a figure explaining the light-dark pattern with respect to a polarizer pattern. It is a figure which shows the relationship between the azimuth angle of a polarizer, and received light intensity. It is explanatory drawing explaining the arrangement | sequence pattern of a parallax pixel. It is a conceptual diagram explaining the relationship between a parallax pixel and a to-be-photographed object. It is a conceptual diagram explaining the process which produces | generates a parallax image. It is a figure explaining the color filter pattern of a Bayer arrangement | sequence and another arrangement | sequence. It is a conceptual diagram which shows the relationship between the output of an image pick-up element, and the produced | generated image data. It is explanatory drawing explaining the 1st arrangement | sequence in an image pick-up element. It is explanatory drawing explaining the 2nd arrangement | sequence in an image pick-up element. It is explanatory drawing explaining the 3rd arrangement | sequence in an image pick-up element. It is explanatory drawing explaining the 4th arrangement | sequence in an image sensor. It is explanatory drawing explaining the 5th arrangement | sequence in an image sensor. It is explanatory drawing explaining the 6th arrangement | sequence in an image sensor. It is explanatory drawing explaining the 7th arrangement | sequence in an image pick-up element. It is a flowchart which shows an outline extraction process. It is a figure which shows the to-be-photographed image observed from an optical finder, and a ranging area.

  Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, not all the combinations of features described in the embodiments are essential for the solving means of the invention.

  The digital camera according to the present embodiment, which is one form of the image pickup apparatus, is configured to generate color image data and polarized image data for one scene by one shooting. The polarization image data is image data representing the polarization state of the reflected light on the subject surface. Further, the digital camera according to the present embodiment can generate parallax image data as an additional function. Each image having a different viewpoint is called a parallax image.

  FIG. 1 is a diagram illustrating the configuration of a digital camera 10 according to an embodiment of the present invention. The digital camera 10 includes a photographic lens 20 as a photographic optical system, and guides a subject light beam incident along the optical axis 21 to the image sensor 100. The photographing lens 20 may be an interchangeable lens that can be attached to and detached from the digital camera 10. The digital camera 10 includes an image sensor 100, a control unit 201, an A / D conversion circuit 202, a memory 203, a drive unit 204, an image processing unit 205, a memory card IF 207, an operation unit 208, a display unit 209, an LCD drive circuit 210, and an AF. A sensor 211 is provided.

  As shown in the figure, the direction parallel to the optical axis 21 toward the image sensor 100 is defined as the z-axis plus direction, the direction toward the front of the drawing in the plane orthogonal to the z-axis is the x-axis plus direction, and the upward direction on the drawing is y. The axis is defined as the plus direction. In the following several figures, the coordinate axes are displayed so that the orientation of each figure can be understood with reference to the coordinate axes of FIG.

  The taking lens 20 is composed of a plurality of optical lens groups, and forms an image of a subject light flux from the scene in the vicinity of its focal plane. In FIG. 1, for convenience of explanation, the photographic lens 20 is represented by a single virtual lens arranged in the vicinity of the pupil. The image sensor 100 is disposed near the focal plane of the photographic lens 20. The image sensor 100 is an image sensor such as a CCD or CMOS sensor in which a plurality of photoelectric conversion elements are two-dimensionally arranged. The image sensor 100 is controlled in timing by the drive unit 204, converts the subject image formed on the light receiving surface into an image signal, and outputs the image signal to the A / D conversion circuit 202.

  The A / D conversion circuit 202 converts the image signal output from the image sensor 100 into a digital image signal and outputs the digital image signal to the memory 203. The image processing unit 205 performs various image processing using the memory 203 as a work space, and generates image data. In particular, the image processing unit 205 includes a contour extraction unit 231 that processes polarized image data and extracts a contour of a subject image. Details of specific processing will be described later.

  In addition, the image processing unit 205 generates color image data and polarized image data from the input image signal in accordance with the pixel arrangement of the image sensor 100, and adjusts the image data according to the selected image format. It also has a function to perform. The generated image data can be converted into a display signal by the LCD driving circuit 210 and displayed on the display unit 209. Further, it can be recorded on the memory card 220 attached to the memory card IF 207.

  The AF sensor 211 of the AF unit is a phase difference sensor in which a plurality of distance measurement areas are set with respect to the subject space, and detects the defocus amount of the subject image in each distance measurement area. A series of shooting sequences is started when the operation unit 208 receives an operation of the user and outputs an operation signal to the control unit 201. Various operations such as AF and AE accompanying the imaging sequence are executed under the control of the control unit 201. For example, the control unit 201 analyzes the detection signal of the AF sensor 211 and executes focus control for moving a focus lens that constitutes a part of the photographing lens 20.

  Next, the configuration of the image sensor 100 will be described in detail. FIG. 2 is a schematic diagram showing a cross section of the image sensor 100 according to the embodiment of the present invention. The image sensor 100 includes a polarizer 101, a color filter 102, an aperture mask 103, a wiring layer 105, and a photoelectric conversion element 108 arranged in this order from the subject side. The photoelectric conversion element 108 is configured by a photodiode that converts incident light into an electrical signal. A plurality of photoelectric conversion elements 108 are two-dimensionally arranged on the surface of the substrate 109.

  An image signal converted by the photoelectric conversion element 108, a control signal for controlling the photoelectric conversion element 108, and the like are transmitted and received through the wiring 106 provided in the wiring layer 105. In addition, an opening mask 103 having openings 104 provided in one-to-one correspondence with each photoelectric conversion element 108 is provided in contact with the wiring layer. As will be described later, the opening 104 is shifted for each corresponding photoelectric conversion element 108 so that the relative position is precisely determined. As will be described in detail later, parallax occurs in the subject light beam received by the photoelectric conversion element 108 by the action of the opening mask 103 including the opening 104.

  On the other hand, the aperture mask 103 does not exist on the photoelectric conversion element 108 that does not generate parallax. In other words, it can be said that the aperture mask 103 having the aperture 104 that does not limit the subject luminous flux incident on the corresponding photoelectric conversion element 108, that is, allows the entire effective luminous flux to pass therethrough is provided. Although no parallax is generated, the aperture 107 formed by the wiring 106 defines the subject luminous flux that is incident, so the wiring 106 is regarded as an aperture mask that allows the entire effective luminous flux that does not cause parallax to pass. You can also. The opening mask 103 may be arranged separately and independently corresponding to each photoelectric conversion element 108, or may be formed collectively for a plurality of photoelectric conversion elements 108 in the same manner as the manufacturing process of the color filter 102. .

  The color filter 102 is provided on the opening mask 103. The color filter 102 is a filter provided in a one-to-one correspondence with each photoelectric conversion element 108, which is colored so as to transmit a specific wavelength band to each photoelectric conversion element 108. In order to output a color image, it is only necessary to arrange at least two types of color filters that are different from each other. However, in order to obtain a higher quality color image, it is preferable to arrange three or more types of color filters. For example, a red filter that transmits a red wavelength band, a green filter that transmits a green wavelength band, and a blue filter that transmits a blue wavelength band may be arranged in a lattice pattern. Further, the corresponding color filter 102 may not be provided in the specific photoelectric conversion element 108. Or you may arrange | position the transparent filter which does not give coloring so that the substantially all wavelength band of visible light may be permeate | transmitted. A specific arrangement will be described later.

  The polarizer 101 is provided on the color filter 102. The polarizer 101 is an optical element that emits an incident subject light flux with different transmittances depending on the polarization direction. The polarizer 101 is provided in one-to-one correspondence with each of the photoelectric conversion elements 108. The polarizer 101 is provided by the corresponding photoelectric conversion element 108 so that the polarization direction to be transmitted is different. Further, depending on the corresponding photoelectric conversion element 108, a polarizer 101 that does not have polarization characteristics in a specific direction is provided. In the present embodiment, even when it does not have a polarization characteristic in a specific direction, it is formed with a polarizer having a polarization characteristic as described below.

  The polarizer 101 is composed of a photonic crystal polarizer. A photonic crystal polarizer is a structure in which optical materials having different refractive indexes are periodically arranged. In particular, when a fine concave-convex groove is provided in the in-plane direction, in-plane anisotropy is generated for incident light. That is, the incident light having the polarization component in the direction along the concave and convex grooves is transmitted better. This direction is called the azimuth angle. When the concave and convex grooves are not provided, the polarizer 101 does not have a polarization characteristic in a specific direction. That is, it has no azimuth.

Specifically, the photonic crystal polarizer is formed by a self-cloning film forming technique or the like using a material that can be formed by sputtering, such as SiO ₂ , Nb ₂ O ₅ , and Al ₂ O ₅ . In particular, the direction of the concavo-convex groove can be changed corresponding to each photoelectric conversion element 108. Moreover, it is also possible not to provide the concave and convex grooves and to give the polarization characteristics in a specific direction. Therefore, the photonic crystal polarizer can be formed integrally with a polarizer 101 having different polarization characteristics and a polarizer 101 having no polarization characteristics in a specific direction.

  As described above, one unit of the aperture mask 103, the color filter 102, and the polarizer 101 provided on a one-to-one basis corresponding to each photoelectric conversion element 108 is referred to as a pixel. In particular, a pixel provided with the opening mask 103 that generates parallax is referred to as a parallax pixel, and a pixel that is not provided with the opening mask 103 that generates parallax is referred to as a non-parallax pixel. A pixel provided with a polarizer 101 having a polarization characteristic in a specific direction is referred to as a polarization pixel, and a pixel provided with a polarizer 101 not having a polarization characteristic in a specific direction is referred to as a non-polarized pixel. For example, when the effective pixel area of the image sensor 100 is about 24 mm × 16 mm, the number of pixels reaches about 12 million.

  A microlens that is a condensing lens for guiding more incident subject light flux to the photoelectric conversion element 108 may be provided in correspondence with each pixel. In consideration of the relative positional relationship between the pupil center of the photographing lens 20 and the photoelectric conversion element 108, the microlens has its optical axis shifted so that more subject light flux is guided to the photoelectric conversion element 108. good. In the case of a back-illuminated image sensor, the wiring layer 105 is provided on the side opposite to the photoelectric conversion element 108.

  The polarizer 101 will be further described. FIG. 3 is a diagram illustrating a light / dark pattern with respect to a polarizer pattern. In particular, FIG. 3A is a diagram showing a polarizer pattern, and FIG. 3B is a diagram conceptually showing the amount of incident subject light flux.

  As described above, the polarizer 101 emits the incident subject light flux with different transmittance depending on the direction of the formed concave and convex grooves. In the present embodiment, as shown in the drawing, four polarizers 101 having azimuth angles in directions of 0 °, 45 °, 90 °, and 135 ° are combined to form a set of polarizer patterns. .

  The subject luminous flux that passes through the polarizer pattern generates various bright and dark patterns. For example, a subject luminous flux composed of linearly polarized light in the 0 ° direction is almost transmitted to the polarizer 101 having an azimuth angle of 0 °, whereas the subject luminous flux is directed to the polarizer 101 having azimuth angles of 45 ° and 135 °. The polarizer 101 having a 90 ° azimuth angle is almost cut off. In addition, the subject luminous flux composed of circularly polarized light is halved with respect to any polarizer 101. That is, the amount of light reaching the photoelectric conversion element 108 varies depending on how much the incident light beam includes the azimuth angle component of the incident polarizer 101.

  FIG. 4 is a diagram illustrating the relationship between the azimuth angle of the polarizer 101 and the amount of light received by the photoelectric conversion element 108. Specifically, the received light amounts of the polarized pixels having azimuth angles of 0 °, 45 °, 90 °, and 135 ° close to each other are shown. Note that these polarization pixels are provided corresponding to the color filters 102 of the same color. The horizontal axis is the azimuth angle of the polarizer 101, and explicitly indicates sampling points of 0 °, 45 °, 90 °, and 135 °. The vertical axis represents the amount of light received by the photoelectric conversion element 108 and, by extension, the amount of charge after photoelectric conversion.

Here, the case that receives an object light beam B ₁ and the subject light flux B ₂ as an example. If the object beam B _1, each polarization pixel, receives the amount of light indicated by white circles. When a cosine curve that smoothly connects the white circles is applied, f ₁ (θ) is obtained. At this time, the maximum value of f ₁ (θ) is F ₁ . Also, if the subject light flux B _2, each polarization pixel, receives the amount of light indicated by black circles. When a cosine curve that smoothly connects the black circles is applied, f ₂ (θ) is obtained. At this time, the maximum value of f ₂ (θ) is F ₂ . Each cosine curve includes polarization information.

  A generalized cosine curve is expressed as f (θ) = A × cos {(π / 2) θ + 2φ} + M. Here, A + M, which is the maximum value of f (θ), can be regarded as the amount of received light when there is no polarizer 101. The polarization state is determined from A, M, and φ. Therefore, as described above, if the polarizer 101 includes the polarizers 101 of 0 °, 45 °, 90 °, and 135 °, the received light amount when the polarizer 101 is not provided from the output of each polarization pixel. Then, the polarization state of the subject luminous flux is determined.

  When the polarization state of the subject luminous flux is determined in this way, the normal direction of the subject surface that is the reflection surface can be estimated. If the outputs of the respective polarization pixels are gathered together, a normal map corresponding to the subject image can be generated. By assigning a color corresponding to the normal direction, a visualized polarization map can also be generated. The polarization map is an image obtained by visualizing the subject image in the normal direction of the subject surface. For example, in a portion where the normal direction is discontinuous, the assigned color changes sharply, and therefore can be recognized as different subjects by a boundary expressed as a sudden change in color. Therefore, a normal map or a polarization map can be used in subject separation in image processing. In the present embodiment, the normal map is handled as polarization image data representing the polarization state of the reflected light on the subject surface.

  Next, the parallax pixel will be described. FIG. 5 is an explanatory diagram for explaining an arrangement pattern of parallax pixels. Here, in order to simplify the explanation, the color arrangement of the color filter 102 is not considered until the reference is resumed later. In the following description that does not refer to the color arrangement of the color filter 102, it can be considered that the image sensor is a collection of only parallax pixels having the color filter 102 of the same color. Therefore, the repetitive pattern described below may be considered as an adjacent pixel in the color filter 102 of the same color. In the present embodiment, the deflection pixels and the parallax pixels are exclusively arranged. That is, a pixel having the opening 104 that causes parallax does not have the polarizer 101 having an azimuth angle.

  As shown in FIG. 5, the opening 104 of the opening mask 103 is provided so as to be shifted relative to the center of each pixel. Even when adjacent pixels are compared, the openings 104 are provided at positions displaced from each other.

  In the example shown in the drawing, six types of opening masks 103 that are shifted in the left-right direction are prepared as the positions of the openings 104 for the respective pixels. The entire image sensor 100 has a two-dimensional and periodic array of photoelectric conversion element groups each including a set of six parallax pixels each having an aperture mask 103 that gradually shifts from the left side to the right side of the drawing. . That is, it can be said that the image sensor 100 is configured by periodically repeating a repeating pattern 110 including a set of photoelectric conversion element groups.

  For example, when one set of three parallax pixels each having three types of opening masks 103 shifted in the left-right direction is set, the opening 104 of the middle pixel does not shift with respect to the center of the pixel. However, also in such a case, parallax images that give parallax to each other can be generated by a method described later, for example, by matching the opening area with the adjacent opening 104. Therefore, the repeating pattern 110 can be defined for combinations corresponding to the odd types of opening masks 103 as described above.

  FIG. 6 is a conceptual diagram illustrating the relationship between the parallax pixels and the subject. In particular, FIG. 6A shows a photoelectric conversion element group of a repetitive pattern 110t arranged at the center orthogonal to the photographing optical axis 21 in the image sensor 100, and FIG. 6B shows a repetitive array arranged in the peripheral portion. The photoelectric conversion element group of the pattern 110u is typically shown. The subject 30 in FIGS. 6A and 6B is in the in-focus position with respect to the photographic lens 20. FIG. 6C schematically shows a relationship when the subject 31 existing at the out-of-focus position with respect to the photographing lens 20 is captured corresponding to FIG.

  First, the relationship between the parallax pixels and the subject when the photographing lens 20 captures the subject 30 that is in focus will be described. The subject luminous flux passes through the pupil of the photographic lens 20 and is guided to the image sensor 100. Six partial areas Pa to Pf are defined for the entire cross-sectional area through which the subject luminous flux passes. For example, in the pixel at the left end of the sheet of the photoelectric conversion element group constituting the repetitive patterns 110t and 110u, only the subject luminous flux emitted from the partial region Pf reaches the photoelectric conversion element 108 as can be seen from the enlarged view. The position of the opening 104f of the opening mask 103 is determined. Similarly, toward the rightmost pixel, the position of the opening 104e corresponding to the partial area Pe, the position of the opening 104d corresponding to the partial area Pd, and the position of the opening 104c corresponding to the partial area Pc. However, the position of the opening 104b is determined corresponding to the partial area Pb, and the position of the opening 104a is determined corresponding to the partial area Pa.

  In other words, the position of the opening 104f is determined by the inclination of the principal ray Rf of the subject light beam emitted from the partial region Pf, which is defined by the relative positional relationship between the partial region Pf and the leftmost pixel, for example. You can say. Then, when the photoelectric conversion element 108 receives the subject luminous flux from the subject 30 existing at the in-focus position via the opening 104f, the subject luminous flux is coupled on the photoelectric conversion element 108 as shown by the dotted line. Image. Similarly, toward the rightmost pixel, the position of the opening 104e is determined by the inclination of the principal ray Re, the position of the opening 104d is determined by the inclination of the principal ray Rd, and the position of the opening 104c is determined by the inclination of the principal ray Rc. It can be said that the position of the opening 104b is determined by the inclination of the light ray Rb, and the position of the opening 104a is determined by the inclination of the principal ray Ra.

  As shown in FIG. 6A, the light beam emitted from the minute region Ot on the subject 30 that intersects the optical axis 21 among the subject 30 existing at the in-focus position passes through the pupil of the photographing lens 20. Then, each pixel of the photoelectric conversion element group constituting the repetitive pattern 110t is reached. That is, each pixel of the photoelectric conversion element group constituting the repetitive pattern 110t receives a light beam emitted from one minute region Ot through each of the six partial regions Pa to Pf. Although the minute region Ot has an extent corresponding to the positional deviation of each pixel of the photoelectric conversion element group constituting the repetitive pattern 110t, it can be approximated to substantially the same object point. Similarly, as shown in FIG. 6B, the light beam emitted from the minute region Ou on the subject 30 that is separated from the optical axis 21 among the subject 30 that exists at the in-focus position passes through the pupil of the photographing lens 20. It passes through and reaches each pixel of the photoelectric conversion element group constituting the repetitive pattern 110u. That is, each pixel of the photoelectric conversion element group constituting the repetitive pattern 110u receives a light beam emitted from one minute region Ou through each of the six partial regions Pa to Pf. Similarly to the micro area Ot, the micro area Ou has an extent corresponding to the positional deviation of each pixel of the photoelectric conversion element group constituting the repetitive pattern 110u, but substantially the same object point. Can be approximated.

  In other words, as long as the subject 30 exists at the in-focus position, the minute area captured by the photoelectric conversion element group differs according to the position of the repetitive pattern 110 on the image sensor 100, and each pixel constituting the photoelectric conversion element group Captures the same minute region through different partial regions. In each repetitive pattern 110, corresponding pixels receive the subject luminous flux from the same partial area. That is, in the figure, for example, the leftmost pixel of each of the repeated patterns 110t and 110u receives the subject light flux from the same partial region Pf.

  In the repetitive pattern 110t arranged in the center orthogonal to the photographing optical axis 21, the left end pixel in the repetitive pattern 110u arranged in the peripheral portion and the position of the opening 104f where the left end pixel receives the subject light beam from the partial region Pf. However, the position of the opening 104f that receives the subject luminous flux from the partial region Pf is strictly different. However, from a functional point of view, these can be treated as the same type of aperture mask in terms of an aperture mask for receiving the subject light flux from the partial region Pf. Therefore, in the example of FIG. 6, it can be said that each of the parallax pixels arranged on the image sensor 100 includes one of six types of aperture masks.

  Next, the relationship between the parallax pixels and the subject when the photographing lens 20 captures the subject 31 existing in the out-of-focus state will be described. Also in this case, the subject luminous flux from the subject 31 present at the out-of-focus position passes through the six partial areas Pa to Pf of the pupil of the photographing lens 20 and reaches the image sensor 100. However, the subject light flux from the subject 31 existing at the out-of-focus position forms an image at another position, not on the photoelectric conversion element 108. For example, as illustrated in FIG. 6C, when the subject 31 exists at a position farther from the imaging element 100 than the subject 30, the subject luminous flux forms an image on the subject 31 side with respect to the photoelectric conversion element 108. Conversely, when the subject 31 is present at a position closer to the image sensor 100 than the subject 30, the subject luminous flux forms an image on the opposite side of the subject 31 from the photoelectric conversion element 108.

Therefore, the subject luminous flux emitted from the minute region Ot ′ among the subjects 31 existing at the out-of-focus position depends on which of the six partial regions Pa to Pf, the corresponding pixels in the different sets of repetitive patterns 110. To reach. For example, as shown in the enlarged view of FIG. 6C, the subject light flux that has passed through the partial region Pd is incident on the photoelectric conversion element 108 having the opening 104d included in the repeated pattern 110t ′ as the principal ray Rd ′. To do. Even if the subject light beam is emitted from the minute region Ot ′, the subject light beam that has passed through another partial region does not enter the photoelectric conversion element 108 included in the repetitive pattern 110t ′, and the repetitive pattern in the other repetitive pattern. The light enters the photoelectric conversion element 108 having a corresponding opening. In other words, the subject luminous flux reaching each photoelectric conversion element 108 constituting the repetitive pattern 110t ′ is a subject luminous flux radiated from different minute areas of the subject 31. That is, a subject luminous flux having a principal ray as Rd ′ is incident on 108 corresponding to the opening 104d, and the principal rays are incident on Ra ⁺ , Rb ⁺ , Rc ⁺ , Re to the photoelectric conversion elements 108 corresponding to the other openings. ^+, although subject light flux to Rf ⁺ is incident, these object light is a subject light flux emitted from different micro region of the object 31. Such a relationship is the same in the repeated pattern 110u arranged in the peripheral portion in FIG.

  Then, when viewed as a whole of the imaging element 100, for example, the subject image A captured by the photoelectric conversion element 108 corresponding to the opening 104a and the subject image D captured by the photoelectric conversion element 108 corresponding to the opening 104d are: If the image is for the subject present at the in-focus position, there is no shift, and if the image is for the subject present at the out-of-focus position, there is a shift. Then, the direction and amount of the shift are determined by how much the subject existing at the out-of-focus position is shifted from the focus position and by the distance between the partial area Pa and the partial area Pd. That is, the subject image A and the subject image D are parallax images. Since this relationship is the same for the other openings, six parallax images are formed corresponding to the openings 104a to 104f.

  Therefore, when the outputs of the pixels corresponding to each other in each of the repetitive patterns 110 configured in this way are collected, a parallax image is obtained. That is, the output of the pixel that has received the subject light beam emitted from a specific partial area among the six partial areas Pa to Pf forms a parallax image.

  FIG. 7 is a conceptual diagram illustrating processing for generating a parallax image. The figure shows, in order from the left column, the generation of the parallax image data Im_f generated by collecting the outputs of the parallax pixels corresponding to the opening 104f, the generation of the parallax image data Im_e by the output of the opening 104e, the opening State of generation of parallax image data Im_d by output of 104d, state of generation of parallax image data Im_c by output of opening 104c, state of generation of parallax image data Im_b by output of opening 104b, parallax by output of opening 104a This represents how the image data Im_a is generated. First, how the parallax image data Im_f is generated by the output of the opening 104f will be described.

  A repeating pattern 110 composed of a group of photoelectric conversion elements each including six parallax pixels is arranged in a horizontal row. Accordingly, the parallax pixels having the opening 104f exist every six pixels in the left-right direction and continuously in the vertical direction on the image sensor 100. Each of these pixels receives the subject luminous flux from different microregions as described above. Therefore, when the outputs of these parallax pixels are collected and arranged, a parallax image is obtained.

  However, since each pixel of the image sensor 100 according to the present embodiment is a square pixel, simply gathering results in the result that the number of pixels in the horizontal direction is reduced to 1/6, and vertically long image data is generated. End up. Therefore, when the aspect ratio in the vertical and horizontal directions is different from the original aspect ratio, an interpolation process is performed. According to the example in the figure, the parallax image data Im_f is generated as an image of the original aspect ratio by performing an interpolation process to make the number of pixels 6 times in the horizontal direction. However, since the parallax image data before the interpolation processing is an image that is thinned by 1/6 in the horizontal direction, the resolution in the horizontal direction is lower than the resolution in the vertical direction. That is, it can be said that the number of generated parallax image data and the improvement in resolution are in a conflicting relationship.

  Similarly, parallax image data Im_e to parallax image data Im_a are obtained. That is, the digital camera 10 can generate a six-view parallax image having parallax in the horizontal direction. In the above example, an example in which a horizontal row is periodically arranged as a repeating pattern 110 has been described, but the repeating pattern 110 is not limited to this. It may be a vertical repeating pattern or a repeating pattern combining oblique directions.

  FIG. 8 is a diagram for explaining the Bayer array and other color filter patterns. FIG. 8A shows a so-called Bayer array, and FIG. 8B shows an example of another color filter array.

  As shown in FIG. 8A, the Bayer array is an array in which the green filter is assigned to the upper left and lower right two pixels, the red filter is assigned to the lower left one pixel, and the blue filter is assigned to the upper right one pixel. Here, the upper left pixel to which the green filter is assigned is the Gb pixel, and the lower right pixel to which the green filter is assigned is the Gr pixel. In addition, a pixel to which a red filter is assigned is an R pixel, and a pixel to which blue is assigned is a B pixel.

  Further, as shown in FIG. 8B, the other color filter array maintains the Gr pixels in the Bayer array shown in FIG. 8A as G pixels to which the green filter is assigned, while the Gb pixels are used as color filters. Is an array changed to W pixels that cannot be assigned. Note that, as described above, the W pixel may be arranged with a transparent filter that is not colored so as to transmit substantially all the wavelength band of visible light. If such a color filter array including W pixels is adopted, the accuracy of the color information output from the image sensor is slightly reduced, but the amount of received light may be increased as compared with the case where a color filter is provided. it can.

  For such an arrangement of the color filter 102, an enormous number of repetitions are performed depending on what color pixels the parallax pixels and non-parallax pixels, and the deflection pixels and non-polarization pixels are allocated to which pixels. A pattern 110 may be set. The image sensor 100 outputs image data having different properties depending on the set repeating pattern 110. FIG. 9 is a conceptual diagram showing the relationship between the output of the image sensor 100 and the generated image data.

  For example, when the effective pixel of the image sensor 100 is 12 million pixels, the pixel output is 12 million. However, the meaning of the output value differs depending on the structure of the pixel output. For example, in the case of an output from a pixel that is an R pixel, a non-polarized pixel, and a non-parallax pixel, the output value represents a single output as a red component of the subject luminous flux. On the other hand, for example, if it is a G pixel, a deflection pixel having an azimuth angle of 45 ° and a pixel without parallax, it represents a superimposed output in which the green component of the subject light beam and the polarization component in the 45 ° direction overlap. Further, for example, if it is a B pixel, a non-polarized pixel, and a parallax pixel, it represents a superimposed output that is a blue component and a specific incident angle component of the subject light flux.

  Therefore, color 2D image data, polarized image data, and parallax image data are obtained from the output from the image sensor by performing processing for separating the superimposed output in accordance with its meaning or interpolating from the output of surrounding pixels. Can be generated. Color 2D image data is image data obtained by capturing a subject image as a normal color image without parallax. As described above, the polarization image data is image data representing the polarization state of the reflected light on the subject surface. In the present embodiment, the normal map is handled as polarization image data. Further, the parallax image data includes, for example, so-called stereoscopic vision, two image data of L image data and R image data. Of course, when the image sensor 100 does not include a parallax pixel, parallax image data is not generated.

  In the repetitive pattern 110, a color 2D image with high resolution can be output by relatively increasing the number of pixels to be output alone. In this case, since the deflection pixels and the parallax pixels have a relatively small ratio, the accuracy of the polarization image data and the image quality of the parallax image are lowered. If many color pixels are used as deflection pixels, the accuracy of the polarization image data increases, but the image quality of the color 2D image decreases as the amount of received light decreases. Further, if the ratio of the parallax pixels is increased, the image quality is improved as the parallax image, but the non-parallax pixels are relatively reduced, so that a 2D image with low resolution is output. In such a trade-off relationship, various pixels may be used depending on which pixel is a deflection pixel or a non-polarized pixel, and which pixel is a parallax pixel or a non-parallax pixel. A repeating pattern 110 having characteristics is set.

  The characteristic arrangement will be sequentially described below. In the following example, when the parallax pixels are arranged, two types of parallax L pixels in which the opening 104 is eccentric to the left side of the pixel center and parallax R pixels that are also eccentric to the right side are assumed. That is, the two viewpoint parallax images output from such parallax pixels realize so-called stereoscopic vision. As described above, when it is desired to output a plurality of parallax images from different viewpoints, the types of parallax pixels may be increased.

  FIG. 10 is an explanatory diagram illustrating a first arrangement in the image sensor 100. The first color filter array is a Bayer array. A polarizer is arranged in any pixel in which the R filter, the G filter, and the B filter are arranged. When attention is paid to the pixel (x, y), it is provided on the same color of (x-1, y + 1), (x + 1, y + 1), (x-1, y-1) (x-1, y + 1). The polarizer of each pixel thus obtained has an azimuth angle of 0 °, 45 °, 90 °, or 135 °. A repeating pattern 110, which is a minimum repeating unit composed of a combination of a color filter array and a polarizer array, is defined by 4 pixels × 4 pixels as shown in the figure. Note that the first array does not include parallax pixels.

  The generation of image data in such an arrangement will be described. The pattern 1101 is a pattern in which four corners centered on the R pixel are B pixels. The B pixels at the four corners are deflection pixels having four kinds of azimuth angles. Accordingly, the blue light quantity value at the coordinates of the center R pixel is calculated from the output of the B pixel at the four corners by the separation algorithm described with reference to FIG. Note that the output of the R pixel is not used to calculate the blue light quantity value at these coordinates.

  The pattern 1102 is a pattern in which the four corners centered on the G pixel are G pixels. In this case as well, as in the case of the pattern 1101, the green light quantity value at the center coordinates is calculated from the output of the four corners. The pattern 1103 is a pattern in which the four corners centered on the B pixel are R pixels. Also in this case, as in the case of the pattern 1101, the red light quantity value at the center coordinates is calculated from the output of the four corners.

  The image data based on the light quantity value of red, green, or blue at each coordinate calculated in this way is image data equivalent to the Bayer array. Therefore, color 2D image data can be generated by performing a lattice interpolation process on the Bayer array on the image data.

  In such an arrangement, polarization image data is also generated in each unit of the patterns 1101, 1102, and 1103. For example, in the pattern 1101, by applying a separation algorithm to the outputs of the B pixels at the four corners, the blue polarization information at the coordinates of the center R pixel can be calculated. If the polarization information corresponding to the Bayer array can be calculated in this way, the polarization image data can be generated by further performing a lattice interpolation process.

  FIG. 11 is an explanatory diagram illustrating a second arrangement in the image sensor. The second color filter array is another color filter array shown in FIG. A polarizer is arranged in the pixel in which the W filter is arranged. Moreover, the parallax L pixel or the parallax R pixel is arranged in the pixel in which the G filter is arranged. Further, as shown by a pattern 1201, when attention is paid to the G pixel of (x, y), (x-1, y + 1), (x + 1, y + 1), (x-1, y-1) (x-1, The polarizer of each pixel of y + 1) has an azimuth angle of 0 °, 45 °, 90 °, or 135 °. A repeating pattern 110 that is a minimum repeating unit composed of a combination of a color filter array, a polarizer array, and a parallax pixel array is defined by 4 pixels × 4 pixels as shown in the figure.

  The generation of image data in such an arrangement will be described. The pattern 1202 is a pattern in which the four corners centered on the W pixel are G pixels. The G pixels at the four corners consist of two pixels of parallax L pixels and two pixels of parallax R pixels. Therefore, the average of these light quantity values is calculated and set as the green light quantity value at the coordinates of the central W pixel. When the green light quantity value at the coordinates of the W pixel is calculated in this way, the average of the green light quantity values at the four corners is further calculated and applied as the green light quantity value of the center pixel in the pattern 1201.

  The image data based on the light quantity value of red, green, or blue at each coordinate calculated in this way is image data equivalent to the Bayer array. Therefore, color 2D image data can be generated by performing a lattice interpolation process on the Bayer array on the image data.

  In such an arrangement, for example, in the pattern 1101, in the pattern 1101, the polarization information at the coordinates of the central G pixel can be calculated by applying a separation algorithm to the outputs of the W pixels at the four corners. In the case of the second arrangement, the polarization pixels are arranged so as to overlap the W pixels, so that highly accurate polarization image data without color component superposition is generated.

  FIG. 12 is an explanatory diagram illustrating a third arrangement in the image sensor. The third color filter array is a Bayer array. A polarizer is arranged in a pixel in which the R filter and the B filter are arranged. When attention is paid to the R pixel or B pixel of (x, y), each pixel of (x-1, y + 1), (x + 1, y + 1), (x-1, y-1) (x-1, y + 1) The polarizer has an azimuth angle of 0 °, 45 °, 90 °, or 135 °. Moreover, the parallax L pixel or the parallax R pixel is arranged in the pixel in which the G filter is arranged. A repeating pattern 110 that is a minimum repeating unit composed of a combination of a color filter array, a polarizer array, and a parallax pixel array is defined by 4 pixels × 4 pixels as shown in the figure.

  The generation of image data in such an arrangement will be described. The pattern 1301 is a pattern in which four corners centered on the R pixel are B pixels. The B pixels at the four corners are deflection pixels having four kinds of azimuth angles. Accordingly, the blue light quantity value at the coordinates of the center R pixel is calculated from the output of the B pixel at the four corners by the separation algorithm described with reference to FIG. Note that the output of the R pixel is not used to calculate the blue light quantity value at these coordinates.

  The pattern 1302 is a pattern in which four corners centering on the B pixel are R pixels. In this case as well, as in the case of the pattern 1301, the red light quantity value at the center coordinates is calculated from the output of the four corners. In addition, since the parallax L pixel and the parallax R pixel adjacent to each other in the upper left and lower right relation are G pixels, the green light quantity value at each coordinate is set as an average value of these two pixels.

  The image data based on the light quantity value of red, green, or blue at each coordinate calculated in this way is image data equivalent to the Bayer array. Therefore, color 2D image data can be generated by performing a lattice interpolation process on the Bayer array on the image data.

  In such an arrangement, polarization image data is also generated in each unit of the patterns 1301 and 1302. For example, in the pattern 1301, by applying a separation algorithm to the outputs of the B pixels at the four corners, the blue polarization information at the coordinates of the center R pixel can be calculated.

  FIG. 13 is an explanatory diagram illustrating a fourth arrangement in the image sensor. The fourth color filter array is a Bayer array. A polarizer is arranged in the pixel where the G filter is arranged. When attention is paid to the G pixel of (x, y), the polarizer of each pixel of (x-1, y + 1), (x + 1, y + 1), (x-1, y-1) (x-1, y + 1) is , 0 °, 45 °, 90 °, or 135 °. Further, the parallax L pixel and the parallax R pixel are arranged in the pixel in which the R filter and the B filter are arranged, respectively. A repeating pattern 110 that is a minimum repeating unit composed of a combination of a color filter array, a polarizer array, and a parallax pixel array is defined by 4 pixels × 4 pixels as shown in the figure.

  The generation of image data in such an arrangement will be described. The pattern 1401 is a pattern in which the four corners centered on the G pixel are G pixels. The G pixels at the four corners are deflection pixels having four kinds of azimuth angles. Therefore, the light quantity value of green at the coordinates of the center G pixel is calculated from the output of the G pixel at the four corners by the separation algorithm described with reference to FIG. Note that the output of the central G pixel is not used to calculate the green light quantity value at this coordinate. Since the R pixel is a parallax pixel, a red light amount value of a pixel without parallax is simply obtained by performing gain adjustment corresponding to the aperture size. Similarly, since the B pixel is also a parallax pixel, the blue light amount value of the pixel without parallax is simply obtained by performing gain adjustment corresponding to the aperture size.

  The image data based on the light quantity value of red, green, or blue at each coordinate calculated in this way is image data equivalent to the Bayer array. Therefore, color 2D image data can be generated by performing a lattice interpolation process on the Bayer array on the image data.

  In such an arrangement, polarization image data is also generated in units of the pattern 1401. For example, in the pattern 1401, by applying a separation algorithm to the output of G pixels at the four corners, green polarization information at the coordinates of the center G pixel can be calculated.

  FIG. 14 is an explanatory diagram illustrating a fifth array in the image sensor. The fifth color filter array is a Bayer array. A polarizer is arranged in any pixel in which the R filter, the G filter, and the B filter are arranged. However, for the G filter, a polarizer is arranged on one of the two G filters in the Bayer arrangement. On the other side where the polarizer is not arranged, a parallax L pixel or a parallax R pixel is arranged. Further, as shown by a pattern 1501, when focusing on the R pixel of (x, y), (x-1, y + 1), (x + 1, y + 1), (x-1, y-1) (x-1, The polarizer of each pixel of y + 1) has an azimuth angle of 0 °, 45 °, 90 °, or 135 °. Similarly, as shown by a pattern 1502, when attention is paid to the G pixel of (x, y), (x-1, y + 1), (x + 1, y + 1), (x-1, y-1) (x-1 , Y + 1), each pixel has a azimuth angle of 0 °, 45 °, 90 °, or 135 °. Similarly, as shown by a pattern 1503, when attention is paid to the B pixel of (x, y), (x-1, y + 1), (x + 1, y + 1), (x-1, y-1) (x-1 , Y + 1), each pixel has a azimuth angle of 0 °, 45 °, 90 °, or 135 °. A repeating pattern 110 that is a minimum repeating unit composed of a combination of a color filter array, a polarizer array, and a parallax pixel array is defined by 4 pixels × 4 pixels as shown in the figure.

  The generation of image data in such an arrangement will be described. The pattern 1501 is a pattern in which the four corners centered on the R pixel are B pixels. The B pixels at the four corners are deflection pixels having four kinds of azimuth angles. Accordingly, the blue light quantity value at the coordinates of the center R pixel is calculated from the output of the B pixel at the four corners by the separation algorithm described with reference to FIG. Note that the output of the R pixel is not used to calculate the blue light quantity value at these coordinates.

  The pattern 1502 is a pattern in which the four corners centered on the G pixel are G pixels. In this case as well, as in the case of the pattern 1501, the green light quantity value at the center coordinates is calculated from the output of the four corners. The pattern 1503 is a pattern in which the four corners centered on the B pixel are R pixels. In this case as well, as in the case of the pattern 1501, the red light amount value at the center coordinates is calculated from the output of the four corners. The pattern 1504 is a pattern in which the four corners centered on the G pixel are G pixels. The G pixels at the four corners consist of two pixels of parallax L pixels and two pixels of parallax R pixels. Therefore, the average of these light quantity values is calculated and set as the green light quantity value at the coordinates of the central G pixel.

  The image data based on the light quantity value of red, green, or blue at each coordinate calculated in this way is image data equivalent to the Bayer array. Therefore, color 2D image data can be generated by performing a lattice interpolation process on the Bayer array on the image data.

  In such an arrangement, polarization image data is also generated in the units of the patterns 1501, 1502, and 1503. For example, in the pattern 1501, by applying a separation algorithm to the outputs of the B pixels at the four corners, it is possible to calculate blue polarization information at the coordinates of the center R pixel.

  FIG. 15 is an explanatory diagram illustrating a sixth arrangement in the image sensor. The sixth array is slightly different from the third array in the arrangement of parallax pixels. Since the generation of the image data is the same as the generation in the third array, the description is omitted.

  FIG. 16 is an explanatory diagram illustrating a seventh arrangement in the image sensor. The seventh color filter array is a Bayer array. A polarizer is arranged on one of the two G filters in the Bayer array. On the other side where the polarizer is not arranged, a parallax L pixel or a parallax R pixel is arranged. As shown by the pattern 1701, when attention is paid to the G pixel of (x, y), (x-1, y + 1), (x + 1, y + 1), (x-1, y-1) (x-1, y + 1) Each pixel has a azimuth angle of 0 °, 45 °, 90 °, or 135 °. Further, the parallax L pixel or the parallax R pixel is also arranged in the pixel in which the R filter and the B filter are arranged. A repeating pattern 110 that is a minimum repeating unit composed of a combination of a color filter array, a polarizer array, and a parallax pixel array is defined by 4 pixels × 4 pixels as shown in the figure.

  The generation of image data in such an arrangement will be described. The pattern 1701 is a pattern in which the four corners centered on the G pixel are G pixels. The G pixels at the four corners are deflection pixels having four kinds of azimuth angles. Therefore, the light quantity value of green at the coordinates of the center G pixel is calculated from the output of the G pixel at the four corners by the separation algorithm described with reference to FIG. Note that the output of the central G pixel is not used to calculate the green light quantity value at this coordinate. The R pixel at (x, y) is a parallax pixel, but the red light quantity at (x, y) is obtained by performing an average process with the output of the R pixels at (x−2, y) and (x + 2, y). Calculate the value. Similarly, the B pixel in (x, y) is a parallax pixel, but by performing the averaging process with the output of the B pixel in (x−2, y) and (x + 2, y), the (x, y) The blue light amount value is calculated. The pattern 1702 is a pattern in which the four corners centered on the G pixel are G pixels. The G pixels at the four corners consist of two pixels of parallax L pixels and two pixels of parallax R pixels. Therefore, the average of these light quantity values is calculated and set as the green light quantity value at the coordinates of the central G pixel.

  The image data based on the light quantity value of red, green, or blue at each coordinate calculated in this way is image data equivalent to the Bayer array. Therefore, color 2D image data can be generated by performing a lattice interpolation process on the Bayer array on the image data.

  In such an arrangement, polarization image data is also generated in units of the pattern 1701. For example, in the pattern 1701, by applying a separation algorithm to the output of G pixels at the four corners, green polarization information at the coordinates of the center G pixel can be calculated.

  Next, image processing in the digital camera 10 including the image sensor 100 as described above will be described. As described above, the digital camera 10 acquires an output signal from the image sensor 100 and generates color 2D image data that is a subject image and polarization image data that represents a polarization state of reflection on the subject surface. Further, if the image sensor 100 includes parallax pixels, parallax image data is also generated. These image processes are executed by the image processing unit 205. Here, image processing for extracting contour information of a subject image using polarization image data in particular will be described. FIG. 17 is a flowchart showing the contour extraction process. The flow starts, for example, from the time when the exposure operation is finished and the image sensor 100 outputs an image signal.

  In step S <b> 101, the image processing unit 205 acquires an output signal of the image sensor 100. In step S102, the image processing unit 205 generates color 2D image data and polarization image data by the above-described method.

  In step S103, the contour extraction unit 231 of the image processing unit 205 performs differentiation processing on the polarization image data. As described above, the polarization image data includes a normal map indicating the normal direction of the subject surface, and the contour extraction unit 231 performs a differentiation process on the normal map. Since the normal direction of the subject surface changes abruptly at the subject boundary, the contour extraction unit 231 can extract the subject boundary line by performing a differentiation process. Proceeding to step S104, the contour extracting unit 231 performs processing for determining the contour of the subject by connecting the extracted boundary lines in consideration of the continuity of each other.

  Proceeding to step S105, the contour extracting unit 231 executes a matching process between the image of the color 2D image data and the image of the polarization image data, and acquires the correspondence between the images. Then, the contour extraction unit 231 proceeds to step S106, applies the contour of the subject determined in step S104 to the color 2D image, determines the contour information in the color 2D image data, and ends the series of processes.

  Note that, according to the above-described flow, the contour extracting unit 231 determines the contour of the subject from the polarization image data, but may be determined in consideration of information on the parallax image data. Since the parallax image data produces a parallax corresponding to the depth difference between the subject images, the contour information of the subject can be obtained similarly by differentiating a so-called depth map that is depth information. Therefore, if the contour information of both is combined, a more accurate subject contour can be determined.

  Next, the relationship between the AF sensor 211 and the image sensor 100 will be described. FIG. 18 is a diagram showing a subject image and a distance measurement area 460 observed from the optical viewfinder. The AF sensor 211 has a plurality of ranging areas 460 that are two-dimensionally and discretely arranged with respect to the subject space. In the case of the example in the figure, eleven ranging areas 460 are discretely arranged in a substantially rhombus shape as a whole. The AF sensor 211 can independently output a defocus amount corresponding to each distance measurement area 460. For example, the control unit 201 detects the defocus amount of the distance measurement area 460 selected by an algorithm such as near point priority, and determines the movement amount and movement direction of the focus lens leading to focusing. Furthermore, the control unit 201 moves the focus lens according to these pieces of information.

  In the above description, the polarizers 101 of the image sensor 100 are evenly arranged with the repeated pattern 110 on a two-dimensional plane. However, considering that an important subject is a subject to be focused, the arrangement region of the polarizer 101 in the image sensor 100 may be limited to a region corresponding to a plurality of distance measurement regions of the AF sensor 211. With this configuration, the number of polarized pixels can be reduced, so that it is possible to improve the image quality of color 2D image data.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above-described embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

  The order of execution of each process such as operations, procedures, steps, and stages in the apparatus, system, program, and method shown in the claims, the description, and the drawings is particularly “before” or “prior to”. It should be noted that the output can be realized in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the description, and the drawings, even if it is described using “first”, “next”, etc. for convenience, it means that it is essential to carry out in this order. It is not a thing.

DESCRIPTION OF SYMBOLS 10 Digital camera, 20 Shooting lens, 21 Optical axis, 30, 31 Subject, 100 Image sensor, 101 Polarizer, 102 Color filter, 103 Aperture mask, 104 Aperture, 105 Wiring layer, 106 Wiring, 107 aperture, 108 Photoelectric conversion Element, 109 substrate, 110 repetitive pattern, 201 control unit, 202 A / D conversion circuit, 203 memory, 204 drive unit, 205 image processing unit, 207 memory card IF, 208 operation unit, 209 display unit, 210 LCD drive circuit, 211 AF sensor, 220 memory card, 231 contour extraction unit, 460 ranging area, 1101, 1102, 1103, 1201, 1202, 1301, 1302, 1401, 1501, 1502, 1503, 1504, 1701, 1702 pattern

Claims (9)

Two-dimensionally arranged photoelectric conversion elements that photoelectrically convert incident light into electrical signals;
A color filter provided in a one-to-one correspondence with each of at least some of the photoelectric conversion elements;
An imaging device comprising: a photonic crystal polarizer provided in a one-to-one correspondence with each of at least some of the photoelectric conversion devices.   The image sensor according to claim 1, wherein the photonic crystal polarizer is provided corresponding to the photoelectric conversion element not provided with the color filter.   The image sensor according to claim 1, wherein the photonic crystal polarizer is provided corresponding to the color filter of the same color.   The photonic crystal in which a group of n photoelectric conversion elements adjacent to each other has a azimuth angle different from at least one color filter pattern of the color filter that transmits different wavelength bands. The imaging device according to any one of claims 1 to 3, wherein the imaging element is periodically arranged to include at least one polarizer pattern formed by combining at least four polarizers. An aperture mask provided in a one-to-one correspondence with each of at least a part of the photoelectric conversion element so as to output at least two parallax image data;
5. The image sensor according to claim 1, wherein the photonic crystal polarizer is provided corresponding to the photoelectric conversion element not provided with the opening mask. 6.   An imaging device provided with the imaging device according to claim 1.   The imaging apparatus according to claim 6, further comprising an image processing unit that obtains an output signal from the imaging element and generates color image data that is a subject image and polarization image data that represents a polarization state of reflection on the subject surface. .   The imaging apparatus according to claim 7, further comprising a contour extraction unit that extracts contour information of the subject image in the color image data based on the polarization image data. An AF unit having a plurality of predetermined ranging areas with respect to the subject space,
The imaging apparatus according to claim 6, wherein the photonic crystal polarizer is provided in a region corresponding to the plurality of distance measuring regions in the imaging device.