JP2016051084A - Imaging device and image pick-up element - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging device and image pick-up element provided in the imaging device that can correctly detect an in-focus state of an edge of a subject.SOLUTION: An imaging device comprises: an image pick-up element that allows a subject to be imaged by incident light to be incident through a focus lens 11; an edge detection unit that detects an edge of the subject imaged on the image pick-up element for each color component; and an image processing engine that functions as an in-focus detection unit detecting an in-focus state of the edge of the subject on the basis of a difference in an amount of color component of the color components in the edge detected by the edge detection unit or ratio of the amounts of color component thereof. An imaging form of the subject relative to the image pick-up element by the incident light is set for each color component of the incident light so that the difference in the amount of color component of the color components in the edge detected by the edge detection unit is broadened.SELECTED DRAWING: Figure 7

Description

  The present invention relates to an imaging device and an imaging element.

  Conventionally, there has been known a focus adjustment device that detects a focus state of a subject edge based on a difference in color component amount of the subject edge detected for each color component from an image and performs focus adjustment based on the detection result. (For example, refer to Patent Document 1).

JP 2011-186252 A

  However, in the conventional focus adjustment device, when the defocus amount of the object edge detected from the image is small, the difference in the color component amount of the edge of each color becomes small, and the focus state of the object edge is accurately determined. There was a problem that it could not be detected.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide an imaging device capable of accurately detecting the focus state of the edge of a subject and an imaging element provided in the imaging device. There is.

  An image pickup apparatus that solves the above problems includes an image pickup device that picks up a subject with incident light incident through a lens, an edge detection unit that detects an edge of the subject picked up by the image pickup device for each color component, and A focus detection unit that detects a focus state of the edge of the subject based on a difference or ratio of color component amounts of the color components at the edge detected by the edge detection unit, and the imaging element using the incident light. The imaging mode of the subject is set for each color component of the incident light so that a difference in color component amount between the color components at the edge detected by the edge detection unit is enlarged.

  In the imaging apparatus, the photoelectric conversion unit of the imaging pixel of the imaging element to which the incident light having a relatively long wavelength side component is incident may be configured such that the incident light having a relatively short wavelength side color component is received. It is preferable to be arranged on the lens side in the optical axis direction of the incident light with respect to the photoelectric conversion unit of the imaging pixel of the incident image sensor.

  In the imaging device, the imaging pixel of the imaging element may have the photoelectric conversion unit in front of the other imaging pixels among the imaging pixels to which the incident light of the color component on the relatively long wavelength side is incident. Compared with the other imaging pixels among the first imaging pixels arranged to protrude toward the lens in the optical axis direction of the incident light and the imaging pixels on which the incident light of the color component on the relatively short wavelength side is incident And it is preferable that the said photoelectric conversion part is comprised including at least one among the 2nd imaging pixels arrange | positioned so that it may become depressed in the optical axis direction of the said incident light on the opposite side to the said lens.

  In addition, it is preferable that the imaging apparatus further includes an image generation unit that generates a captured image using image data captured by at least one of the first imaging pixel and the second imaging pixel.

  In the imaging device, the image generation unit may be configured to obtain image data captured by another imaging pixel in the imaging element with respect to image data captured by at least one of the first imaging pixel and the second imaging pixel. Preferably, the captured image is generated after performing different image processing.

  Further, in the imaging apparatus, the image generation unit uses a shading correction process that uses a shading coefficient that is different from the image data captured by the other imaging pixels of the imaging element for the image data captured by the second imaging pixel. It is preferable to generate the captured image after performing the above as the image processing.

  In the imaging apparatus, the image generation unit performs a magnification correction process on the image data captured by at least one of the first imaging pixel and the second imaging pixel as the image processing, and then performs the imaging. Preferably, an image is generated.

  In the imaging apparatus, it is preferable that at least one of the first imaging pixel and the second imaging pixel is irregularly arranged on the imaging pixels of the imaging element arranged two-dimensionally.

  The imaging apparatus further includes a display control unit that causes the display unit to display an image captured by an imaging pixel including at least one of the first imaging pixel and the second imaging pixel in the imaging element as a display image. Is preferred.

  In the imaging apparatus, the display image is a through image or a moving image, and the display control unit captures an imaging pixel including at least one of the first imaging pixel and the second imaging pixel in the imaging element. It is preferable to display the display image including the processed images as time-sequential continuous frame images on the display unit.

  In the imaging apparatus, the imaging pixel of the imaging element includes a condenser lens that condenses the incident light on a photoelectric conversion unit, and the incident light of a color component on a relatively short wavelength side is incident on the imaging pixel. It is preferable that the focal length of the condensing lens of the imaging pixel is smaller than the focal length of the condensing lens of the imaging pixel on which the incident light of the color component on the relatively long wavelength side is incident.

  The imaging apparatus further includes an element driving unit that drives the imaging element in an optical axis direction of the incident light, and the element driving unit is configured to perform the matching by the incident light of a color component on a relatively short wavelength side. When the subject for detecting a focus state is imaged by the imaging element, the imaging element is arranged at the first position in the optical axis direction of the incident light, while the color component on the relatively long wavelength side When the subject for detecting the in-focus state of the edge by the incident light is imaged by the imaging element, the lens is moved in the optical axis direction of the incident light from the first position. It is preferable to arrange in the second position located on the side.

  An imaging device that solves the above problem includes an imaging pixel in which incident light of each color component is converted into an electric charge, and the imaging pixel receives the incident light of a relatively long wavelength side color component. The imaging pixel is arranged closer to the incident side in the optical axis direction of the incident light than the imaging pixel into which the incident light of the color component on the relatively short wavelength side is incident.

  According to the present invention, it is possible to accurately detect the focus state of the edge of the subject.

1 is a block diagram of a digital camera according to a first embodiment. The schematic diagram which shows the arrangement | sequence of the imaging pixel in the image pick-up element of the embodiment. (A), (b) is a schematic diagram which shows the cross-sectional structure which cut | disconnected the image pick-up element of the embodiment in the optical axis direction. The schematic diagram which shows the area where the pixel group containing a convex pixel and a concave pixel is arrange | positioned among the imaging surfaces of an image pick-up element. (A) is a schematic diagram showing the positional relationship between the imaging element and the focus lens in a state where an image captured by the imaging element is in focus on the near point side of the subject, and (b) is an imaging The schematic diagram which shows the positional relationship of an image pick-up element and a focus lens in the state in which the image imaged by the element is focusing on the far point side rather than the position of a to-be-photographed object. The schematic diagram which shows the positional relationship in the optical axis direction of the focus for every color component of the incident light which injected into the focus lens. It is a schematic diagram which shows the positional relationship of the photoelectric conversion surface for every color component of the incident light which injects from a to-be-photographed object through a focus lens, Comprising: (a) is an image imaged by the image pick-up element in the front pin state with respect to a to-be-photographed object. FIG. 6B is a schematic diagram in a case where an image captured by the image sensor is in a rear pin state with respect to a subject. The schematic diagram which shows an example of the image imaged by the image pick-up element. FIG. 7 is a graph showing a normalized output of color component amounts for each color component in the vicinity of an edge when the image captured by the image sensor is in a front pin state with respect to the edge of the subject, (a) is a comparison The graph in the case of the image sensor of an example, (b) is the graph in the case of the image sensor of 1st Embodiment. 6 is a graph showing a normalized output of color component amounts for each color component in the vicinity of an edge when an image captured by an image sensor is in a back-pin state with respect to the edge of the subject, (a) is a comparison The graph in the case of the image sensor of an example, (b) is the graph in the case of the image sensor of 1st Embodiment. 6 is a graph showing a normalized output of the color component amount for each color component in the vicinity of the edge, where (a) is a state in which the image captured by the image sensor of the first embodiment is in a front pin state with respect to the edge of the subject Alternatively, a graph showing the normalized output of the color component amounts of red light and green light when in the rear pin state, (b) is an image captured by the image sensor of the first embodiment at the edge of the subject The graph which shows the normalized output of the color component amount of blue light and green light in the case of being in the front pin state or the back pin state. 7 is a graph showing a difference between color component amounts of red light and green light in the vicinity of an edge, where (a) is a graph in the case of an image sensor of a comparative example, and (b) is a graph of the first embodiment. The graph in the case of an image sensor. (A) is a graph showing LSF at an in-focus edge, (b) is a graph showing LSF at an out-of-focus edge, and (c) is more defocused than in the state shown in FIG. 13 (b). The graph which shows LSF in the edge where quantity is large. (A) is a graph showing LSF for each color component at the edge in the focused state, (b) is a graph showing LSF for each color component at the edge in the front pin state, and (c) is an edge in the rear pin state. The graph which shows LSF for every color component in. The graph which shows the correlation of the difference of the standard deviation of LSF for every color component in an edge, and object distance. 6 is a flowchart of an image generation processing routine executed by the image processing engine of the first embodiment. (A), (b) is a schematic diagram which shows an example of the structure aspect of the frame image in a through image. The graph which shows the correlation with the shading coefficient in a concave pixel, and an aperture value used as the criterion of whether to use the pixel signal of the to-be-photographed object for a picked-up image. (A), (b) is a schematic diagram which shows the cross-sectional structure which cut | disconnected the image pick-up element of 2nd Embodiment in the optical axis direction. FIG. 6 is a schematic diagram showing a positional relationship for each color component of incident light incident on a photoelectric conversion unit from a subject through a microlens in the image sensor of the embodiment, where (a) shows an image captured by the image sensor. FIG. 5B is a schematic diagram when the subject is in the front pin state, and FIG. 5B is a schematic diagram when the image captured by the image sensor is in the rear pin state with respect to the subject. FIG. 10 is a schematic diagram illustrating a positional relationship of imaging positions for each color component of incident light incident from a subject through a focus lens in the imaging device of the third embodiment, and (a) is an image captured by the imaging device. FIG. 6B is a schematic diagram when the image is captured in the front pin state with respect to the subject, and FIG. (A)-(c) is a schematic diagram which shows the cross-sectional structure which cut | disconnected the white pixel in the optical axis direction, Comprising: (a) is a schematic diagram which shows the image pick-up element of a comparative example, (b) is 4th. The schematic diagram which shows an example of the image pick-up element of embodiment of this, (c) is a schematic diagram which shows another example of the image pick-up element of 4th Embodiment. FIG. 6 is a schematic diagram illustrating a positional relationship of the photoelectric conversion surface for each color component of incident light incident from a subject through a microlens when an image captured by the imaging element is in a front pin state with respect to the subject, FIG. 22A is a schematic diagram in the case of the imaging element of the comparative example shown in FIG. 22A, and FIG. 22B is a schematic diagram in the case of the example of the imaging element of the fourth embodiment shown in FIG. (C) is a schematic diagram in the case of another example of the imaging device of 4th Embodiment shown in FIG.22 (c). FIG. 6 is a schematic diagram showing a positional relationship of the photoelectric conversion surface for each color component of incident light incident from a subject through a microlens when an image captured by the imaging element is in a rear pin state with respect to the subject, FIG. 22A is a schematic diagram in the case of the imaging element of the comparative example shown in FIG. 22A, and FIG. 22B is a schematic diagram in the case of the example of the imaging element of the fourth embodiment shown in FIG. (C) is a schematic diagram in the case of another example of the imaging device of 4th Embodiment shown in FIG.22 (c). (A) is a schematic diagram which shows the area where the pixel group containing a convex pixel and a concave pixel is arrange | positioned among the imaging surfaces of the image pick-up element of another embodiment, (b) is in the image pick-up element of another embodiment. The schematic diagram which shows the arrangement | sequence of an imaging pixel.

(First embodiment)
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings.
As shown in FIG. 1, a digital camera (hereinafter referred to as “camera 10”), which is a kind of imaging device, includes a lens unit 12 (in FIG. 1, a focus lens) including a plurality of lenses such as a focus lens 11 for focus adjustment. 11), and an image sensor 13 that picks up an image by imaging the light passing through the lens unit 12 on the image space side of the lens unit 12. Note that the image sensor 13 is a CMOS (Complementary Metal Oxide Semiconductor) type or CCD (Charge Coupled Device) type image sensor.

  An image processing engine 14 is connected to the output side of the image sensor 13 via an A / D conversion circuit 15. The pixel signal output as an analog signal from the image sensor 13 is converted into a digital signal by the A / D conversion circuit 15 and then input to the image processing engine 14.

  The image processing engine 14 includes an MPU 16 (Micro Processing Unit) that comprehensively controls various operations of the camera 10. Then, the MPU 16 generates a predetermined image by performing image processing such as color interpolation processing, gradation correction, white balance processing, and contour compensation on the pixel signal read from the image sensor 13.

  A non-volatile memory 18, a buffer memory 19, an interface unit (hereinafter referred to as “I / F unit 20”), and a monitor 21 as an example of a display unit are connected to the image processing engine 14 via a data bus 17. ing.

  The nonvolatile memory 18 stores a program executed by the MPU 16 for operating the image processing engine 14. In the present embodiment, the nonvolatile memory 18 stores an image generation processing program shown in the flowchart in FIG. The MPU 16 functions as the edge detection unit 22, the focus detection unit 23, the image generation unit 24, and the display control unit 25 by executing the image generation processing program stored in the nonvolatile memory 18.

The buffer memory 19 temporarily stores, for example, photographed images, images in the image processing process, images after image processing, images after image compression, and the like.
The I / F unit 20 has a card slot (not shown) in which the memory card 30 is detachably mounted. Then, the I / F unit 20 outputs the image generated by the image processing engine 14 to the memory card 30 attached to the I / F unit 20 or the image stored in the memory card 30 as the image processing engine 14. Or has a function to output to

  On the monitor 21, an image temporarily stored in the buffer memory 19 and an image stored in the memory card 30 attached to the I / F unit 20 are output and displayed by the image processing engine 14.

  In addition, a release button 31 and an operation unit 32 are connected to the image processing engine 14. The release button 31 inputs an operation signal to the image processing engine 14 when a half-press operation or a full-press operation is performed. The operation unit 32 includes a menu button, a select button, a determination button, a power button, and the like, and inputs an operation signal to the image processing engine 14 when a pressing operation is performed.

Next, the configuration of the image sensor 13 will be described.
As shown in FIGS. 2 and 3A and 3B, the imaging device 13 has imaging pixels 40 arranged two-dimensionally on the imaging surface. The imaging pixel 40 is configured by a microlens 41 as an example of a condensing lens, a color filter 42, and a photoelectric conversion unit 43. The imaging pixel 40 includes a red pixel 40R (R), a green pixel 40G (G), and a blue pixel 40B (B).

  As shown in FIG. 3A, the red pixel 40R includes a color filter 42R that selectively transmits light in the red wavelength region. Of the incident light incident on the image sensor 13, the light of the red color component that has passed through the color filter 42R is condensed on the photoelectric conversion unit 43R by the microlens 41R, and converted into electric charge in the photoelectric conversion unit 43R. Note that a part of the red pixel 40R constituting the imaging pixel 40 is a convex portion in which the photoelectric conversion unit 43R protrudes toward the incident side (the focus lens 11 side) in the optical axis direction of incident light as compared with the other red pixels 40R. This is a pixel 40RA.

  As shown in FIGS. 3A and 3B, the green pixel 40G has a color filter 42G that selectively transmits light in the green wavelength region. Of the incident light incident on the image sensor 13, light of the green color component that has passed through the color filter 42G is condensed on the photoelectric conversion unit 43G by the microlens 41G, and converted into electric charge by the photoelectric conversion unit 43G.

  As shown in FIG. 3B, the blue pixel 40B has a color filter 42B that selectively transmits light in the blue wavelength region. Of the incident light incident on the image sensor 13, the light of the blue color component that has passed through the color filter 42 </ b> B is condensed on the photoelectric conversion unit 43 </ b> B by the microlens 41 </ b> B and converted into electric charge in the photoelectric conversion unit 43 </ b> B. Note that a part of the blue pixels 40B constituting the imaging pixel 40 is on the emission side in the optical axis direction of the incident light (on the opposite side to the focus lens 11) as compared with the other blue pixels 40B. It is a recessed concave pixel 40BA.

  The photoelectric conversion surface of the photoelectric conversion unit 43R in the convex pixel 40RA is disposed on the incident side (focus lens 11 side) in the optical axis direction of incident light with respect to the photoelectric conversion surface of the photoelectric conversion unit 43 in the flat pixel. In addition, the photoelectric conversion surface of the photoelectric conversion unit 43B in the concave pixel 40BA is disposed on the emission side in the optical axis direction of incident light (the side opposite to the focus lens 11) with respect to the photoelectric conversion surface of the photoelectric conversion unit 43 in the flat pixel. ing. In this case, a flat pixel means a red pixel 40R, a green pixel 40G that is not a convex pixel 40RA, and a blue pixel 40B that is not a concave pixel 40BA.

  In the present embodiment, the red pixel 40R, the green pixel 40G, and the blue pixel 40B have the same curvature of the lens surface of the microlens 41. Therefore, the focal lengths of the microlenses 41 corresponding to the light components of the respective colors out of the incident light incident on the image sensor 13 are equal to each other.

  As shown in FIG. 2, on the image sensor 13, red pixels 40R, green pixels 40G, and blue pixels 40B are arranged in a Bayer Arrangement arrangement pattern. That is, two green pixels 40G are arranged on one diagonal line, and one red pixel 40R and one blue pixel 40B are arranged on the other diagonal line in a pixel group consisting of four adjacent square lattice pixels. . The imaging element 13 is configured by repeatedly arranging the pixel groups in two dimensions with a square lattice pixel group as a basic unit. In the present embodiment, some of the pixel groups constituting the image sensor 13 include a red pixel 40R that is a convex pixel 40RA and a blue pixel 40B that is a concave pixel 40BA. Specifically, a red pixel 40R, which is a convex pixel 40RA, every two pixel groups in the vertical direction of the image pickup surface of the image pickup device 13, and every other pixel group in the horizontal direction of the image pickup surface of the image pickup device 13, and A pixel group including the blue pixel 40B that is the concave pixel 40BA is arranged.

  In this case, as shown in FIG. 4, in the present embodiment, the red pixel 40R that is the convex pixel 40RA and the blue pixel 40B that is the concave pixel 40BA are on the imaging surface among the imaging pixels 40 of the imaging element 13. It is included in a pixel group located in six areas R1 to R6 spaced apart in the vertical direction. Further, these areas R1 to R6 extend over the entire area in the lateral direction on the imaging surface of the imaging element 13.

  Next, an outline of the principle by which the image processing engine 14 detects the in-focus state of the object edge by analyzing the color shift of the longitudinal chromatic aberration at the object edge detected from the image will be described.

  In FIG. 5A, a state where the image captured by the image sensor 13 is in focus at the position of the subject S is indicated by a solid line, and the image captured by the image sensor 13 is more than the position of the subject S. A state of focusing on the near point side (hereinafter referred to as “front pin state”) is indicated by a one-dot chain line. As shown in FIG. 5A, in the front pin state, the position where the light emitted from the subject passes through the focus lens 11 and is in focus is closer to the photographer side than the image sensor 13 (in FIG. 5A). Located on the right).

  On the other hand, in FIG. 5B, a state where the image captured by the image sensor 13 is in focus at the position of the subject S is indicated by a solid line, and the image captured by the image sensor 13 is the position of the subject S. A state of focusing on the far point side (hereinafter referred to as “rear pin state”) is indicated by a one-dot chain line. As shown in FIG. 5B, in the rear pin state, the position where the light emitted from the subject passes through the focus lens 11 and is focused is closer to the focus lens 11 than the image sensor 13 (FIG. 5B). It is located on the left side.

  FIG. 6 shows the state of axial chromatic aberration that occurs when the light emitted from the subject passes through the focus lens 11. As shown in FIG. 6, since axial chromatic aberration occurs in the light that has passed through the focus lens 11 due to the difference in wavelength, the focal position shifts for each color component of the light. Specifically, since the refractive index of light increases as the wavelength of light is shorter, the focus lens 11 is in the order of blue (B) light, green (B) light, and red (R) light. The refractive index when passing through the lens gradually decreases, and the focal position through which the focus lens 11 is focused is gradually distant from the focus lens 11. Then, by analyzing the color shift due to such axial chromatic aberration, the defocus feature amount of the subject is calculated (detected) as the focused state of the subject.

  Here, the defocus feature amount includes a direction index and a defocus amount. The direction index is an index indicating that the image captured by the image sensor 13 is in the front pin state or the rear pin state with respect to the subject. As examples of evaluation values for calculating the direction index and the defocus amount, Edge Difference (hereinafter referred to as “Ed”), defocus amount reference value (Width of Subtraction) (hereinafter referred to as “Wd”), Line Spread Function (hereinafter referred to as “LSF”).

First, the outline of the principle of calculating the direction index and the defocus amount in the subject using Ed will be described.
As shown in FIG. 7A, when the image picked up by the image pickup device 13 is in the front pin state with respect to the subject, the light emitted from the subject is an axis generated when passing through the focus lens 11. Due to the upper chromatic aberration, red (R) light, green (G) light, and blue (B) light are guided from the outside to the image sensor 13 in a state of being dispersed in order. In this case, the defocus amount of the subject that the light passing through the focus lens 11 captures on the image sensor 13 for each color component is in the order of blue (B) light, green (G) light, and red (R) light. It grows gradually.

  In particular, in the present embodiment, the photoelectric conversion surface of the red pixel 40R that is the convex pixel 40RA is arranged at a position closer to the focus lens 11 in the optical axis direction than the photoelectric conversion surface of the green pixel 40G. Therefore, the defocus amount of the subject in which red (R) light is imaged on the photoelectric conversion surface of the red pixel 40R which is the convex pixel 40RA is the photoelectric conversion surface of the red pixel 40R where the red (R) light is a flat pixel. This is larger than the defocus amount of the subject to be imaged. As a result, the defocus amount of the subject in which red (R) light is imaged on the photoelectric conversion surface of the red pixel 40R, which is the convex pixel 40RA, and the subject in which green (G) light is imaged on the photoelectric conversion surface of the green pixel 40G. The difference from the defocus amount becomes larger compared to the case where the red pixel 40R is a flat pixel.

  Similarly, in the present embodiment, the photoelectric conversion surface of the blue pixel 40B that is the concave pixel 40BA is disposed at a position farther from the focus lens 11 in the optical axis direction than the photoelectric conversion surface of the green pixel 40G. For this reason, the defocus amount of the subject in which blue (B) light is imaged on the photoelectric conversion surface of the blue pixel 40B, which is the concave pixel 40BA, is the photoelectric conversion surface of the blue pixel 40B, in which the blue (B) light is a flat pixel. It becomes smaller than the defocus amount of the subject to be imaged. As a result, the defocus amount of the subject in which blue (B) light is imaged on the photoelectric conversion surface of the blue pixel 40B, which is the concave pixel 40BA, and the subject in which green (G) light is imaged on the photoelectric conversion surface of the green pixel 40G. The difference from the defocus amount becomes larger compared to the case where the blue pixel 40B is a flat pixel.

  On the other hand, as shown in FIG. 7B, when the image picked up by the image pickup device 13 is in a back-pin state with respect to the subject, the light emitted from the subject passes through the focus lens 11. Due to the generated axial chromatic aberration, red (R) light, green (G) light, and blue (B) light are guided from the outside to the image sensor 13 in a state of being dispersed in order. In this case, the defocus amount of the subject that the light passing through the focus lens 11 images on the image sensor 13 for each color component is in the order of red (R) light, green (G) light, and blue (B) light. It grows gradually.

  In particular, in the present embodiment, the photoelectric conversion surface of the red pixel 40R that is the convex pixel 40RA is arranged at a position closer to the focus lens 11 in the optical axis direction than the photoelectric conversion surface of the green pixel 40G. Therefore, the defocus amount of the subject in which red (R) light is imaged on the photoelectric conversion surface of the red pixel 40R which is the convex pixel 40RA is the photoelectric conversion surface of the red pixel 40R where the red (R) light is a flat pixel. It becomes smaller than the defocus amount of the subject to be imaged. As a result, the defocus amount of the subject in which red (R) light is imaged on the photoelectric conversion surface of the red pixel 40R, which is the convex pixel 40RA, and the subject in which green (G) light is imaged on the photoelectric conversion surface of the green pixel 40G. The difference from the defocus amount becomes larger compared to the case where the red pixel 40R is a flat pixel.

  Similarly, in the present embodiment, the photoelectric conversion surface of the blue pixel 40B that is the concave pixel 40BA is disposed at a position farther from the focus lens 11 in the optical axis direction than the photoelectric conversion surface of the green pixel 40G. For this reason, the defocus amount of the subject in which blue (B) light is imaged on the photoelectric conversion surface of the blue pixel 40B, which is the concave pixel 40BA, is the photoelectric conversion surface of the blue pixel 40B, in which the blue (B) light is a flat pixel. This is larger than the defocus amount of the subject to be imaged. As a result, the defocus amount of the subject in which blue (B) light is imaged on the photoelectric conversion surface of the blue pixel 40B, which is the concave pixel 40BA, and the subject in which green (G) light is imaged on the photoelectric conversion surface of the green pixel 40G. The difference from the defocus amount becomes larger compared to the case where the blue pixel 40B is a flat pixel.

  That is, depending on whether the image captured by the image sensor 13 is in the front pin state or the rear pin state with respect to the subject, light passing through the focus lens 11 is input to the image sensor 13 for each color component. The magnitude relationship of the defocus amount of the subject to be imaged is different.

  In addition, when the red pixel 40R is the convex pixel 40RA and when the blue pixel 40B is the concave pixel 40BA, the image captured by the image sensor 13 is in the front pin state and the rear pin state with respect to the subject. Regardless of the state, the difference in the defocus amount of the subject that the light passing through the focus lens 11 images on the image sensor 13 for each color component becomes large.

  FIG. 8 shows a black and white chart S <b> 1 as an example of a subject in an image captured by the image sensor 13. In FIG. 8, the X coordinate is set in the horizontal direction and the Y coordinate is set in the vertical direction, and the edge E1 of the black and white chart S1 located at the position of X = q is set as the defocus feature amount calculation target. .

  In this case, first, pixel values for each RGB are obtained in the direction crossing the edge E1 (X direction in FIG. 8). At this time, if a pixel value of only one pixel row is acquired, an erroneous pixel value may be acquired if the one pixel row includes noise. Therefore, in this embodiment, as shown in the following formulas (1) to (3), n pixel values in one pixel row are integrated over the width of n pixels in the Y direction, and the integrated value is divided by n. The average value thus obtained is acquired as a pixel value for each RGB crossed by the edge E1 located at x = q.

  Here, r (x, k), g (x, k), and b (x, k) are n (k = 1, 2,..., N) one pixel row in the n pixel width in the y direction. R pixel value, G pixel value, and B pixel value are shown. In the example shown in FIG. 8, x represents a value having a predetermined range including X = q (for example, q−Q / 2 ≦ x ≦ q + Q / 2) as a domain.

  Subsequently, by normalizing r_ave (x), g_ave (x), and b_ave (x) using respective maximum values and minimum values as shown in the following equations (4) to (6), Each normalized output is calculated.

  Here, xmin represents an X coordinate value at which the pixel value is minimum (black area in the example shown in FIG. 8) in the detection target area EA of the edge E1, and xmax is a pixel in the detection target area EA of the edge E1. The X coordinate value having the maximum value (white area in the example shown in FIG. 8) is shown.

  In FIG. 9A, in the imaging element 13 of the comparative example in which the imaging pixel 40 includes only the flat pixels without including the convex pixel 40RA and the concave pixel 40BA, an image captured by the imaging element 13 is a black and white chart S1. The normalized output of each color component in the vicinity of the edge E1 of the black-and-white chart S1 in the case of the front pin state with respect to the edge E1 is shown. As shown in FIG. 9A, when the image captured by the image sensor 13 is in the front pin state with respect to the edge E1 of the black and white chart S1, blue (B), green (G), red (R ), The gradient of the color component amount at the edge E1 of the black and white chart S1 becomes gradually gentler. This is because when the image captured by the image sensor 13 is in the front pin state with respect to the edge E1 of the black and white chart S1, the black and white chart in which the light passing through the focus lens 11 is captured by the image sensor 13 for each color component. This is because the defocus amount of the edge E1 of S1 gradually increases in the order of blue (B) light, green (G) light, and red (R) light. FIG. 9A shows a state in which the defocus amount of the edge E1 of the black and white chart S1 in the image picked up by the image sensor 13 is small, and each color component at the edge E1 of the black and white chart S1 is shown. The gradient of the color component amount is almost the same.

  On the other hand, in FIG. 9B, in the imaging device 13 of the present embodiment in which the imaging pixel 40 includes the convex pixel 40RA and the concave pixel 40BA, an image captured by the imaging device 13 corresponds to the edge E1 of the black and white chart S1. The normalized output of each color component in the vicinity of the edge E1 of the black and white chart S1 in the case of the front pin state is shown. As shown in FIG. 9B, in the image sensor 13 of the present embodiment, the focus lens 11 even if the defocus amount of the edge E1 of the black and white chart S1 in the image captured by the image sensor 13 is small. The difference in the defocus amount of the edge E1 of the black and white chart S1 in which the light that has passed through the image pickup device 13 is imaged for each color component is increased. Therefore, the gradient of the color component amount of each color component at the edge E1 of the black-and-white chart S1 is greatly different from that in the case where the imaging element 13 of the comparative example shown in FIG. 9A is used.

  FIG. 10A shows the edge E1 of the black-and-white chart S1 when the image picked up by the image-capturing element 13 is in a rear pin state with respect to the edge E1 of the black-and-white chart S1. The normalized output of each color component in the vicinity of is shown. As shown in FIG. 10A, when the image picked up by the image pickup device 13 is in a rear pin state with respect to the edge E1 of the black and white chart S1, red (R), green (G), blue (B ), The gradient of the color component amount at the edge E1 of the black and white chart S1 becomes gradually gentler. This is because the image captured by the image sensor 13 is in a back-pin state with respect to the edge E1 of the black and white chart S1, and the black and white chart in which the light passing through the focus lens 11 is captured by the image sensor 13 for each color component. This is because the defocus amount of the edge E1 of S1 gradually increases in the order of red (R) light, green (G) light, and blue (B) light. FIG. 10A shows a state in which the defocus amount of the edge E1 of the black and white chart S1 in the image picked up by the image sensor 13 is small, and each color component at the edge E1 of the black and white chart S1 is shown. The gradient of the color component amount is almost the same.

  On the other hand, FIG. 10B shows the edge of the black and white chart S1 when the image picked up by the image pickup element 13 is in a back-pin state with respect to the edge E1 of the black and white chart S1. The normalized output of each color component in the vicinity of E1 is shown. As shown in FIG. 10B, in the image sensor 13 of the present embodiment, even when the defocus amount of the edge E1 of the black and white chart S1 in the image captured by the image sensor 13 is small, the focus lens 11 The difference in the defocus amount of the edge E1 of the black and white chart S1 in which the light that has passed through the image pickup device 13 is imaged for each color component is increased. Therefore, the gradient of the color component amount of each color component at the edge E1 of the black-and-white chart S1 is greatly different from that in the case where the imaging element 13 of the comparative example shown in FIG.

  That is, as shown in FIG. 11A, in the present embodiment, the vicinity of the edge E1 of the black and white chart S1 when the defocus amount of the edge E1 of the black and white chart S1 in the image picked up by the image sensor 13 is small. The gradient of each color component amount at is as follows. When the image captured by the image sensor 13 is in the front pin state with respect to the edge E1 of the black and white chart S1, the gradient of the red (R) color component amount is greater than the gradient of the green (G) color component amount. Get smaller. On the other hand, when the image picked up by the image pickup device 13 is in the back-pin state with respect to the edge E1 of the black and white chart S1, the gradient of the red (R) color component amount is the green (G) color component amount. It becomes larger than the gradient. Therefore, red (R) at the edge E1 of the black and white chart S1 depends on whether the image captured by the imaging device 13 is in the front pin state or the rear pin state with respect to the edge E1 of the black and white chart S1. The magnitude relationship between the gradient of the color component amount and the gradient of the green (G) color component amount is reversed.

  Therefore, by comparing Ed (hereinafter referred to as “EdRG”) based on the red (R) color component amount and the green (G) color component amount with a threshold value, an image captured by the image sensor 13 is a black and white chart. Whether the state is the front pin state or the rear pin state with respect to the edge E1 of S1 is determined based on the following expressions (7) and (8).

  Here, Σ (R / G) is a sum obtained by adding the value obtained by dividing the red (R) color component amount by the green (G) color component amount at the same position in the image over the section Δ1. Show. Then, a value obtained by dividing the sum by the length of the section Δ1 is calculated as EdRG. The section Δ1 is the color of green (G) in the section located on the right side of the intersection P1 where the red (R) color component amount and the green (G) color component amount intersect in FIG. It is defined as a part of a section where the component amount has a gradient.

  And when Formula (7) is materialized, EdRG has shown that the image imaged by the image pick-up element 13 is a back pin state with respect to the edge E1 of the black-and-white chart S1. On the other hand, when the formula (8) is established, EdRG indicates that the image captured by the image sensor 13 is in the front pin state with respect to the edge E1 of the black and white chart S1. Also, the difference between EdRG and the threshold value “1” indicates the defocus amount of the edge E1 of the black and white chart S1 in the image captured by the image sensor 13.

  It should be noted that whether the image picked up by the image pickup device 13 is in the front pin state or the rear pin state with respect to the edge E1 of the black and white chart S1 is expressed by the following equations (9) and (10). It may be determined based on.

  Here, Σ (R / G) is a sum obtained by adding the value obtained by dividing the red (R) color component amount by the green (G) color component amount at the same position in the image over the section Δ2. Show. Then, a value obtained by dividing the sum by the length of the section Δ2 is calculated as EdRG. The section Δ2 is the green (G) color in the section located on the left side of the intersection P1 where the red (R) color component amount and the green (G) color component amount intersect in FIG. It is defined as a part of a section where the component amount has a gradient.

  And when Formula (9) is materialized, EdRG has shown that the image imaged by the image pick-up element 13 is a front pin state with respect to the edge E1 of the black-and-white chart S1. On the other hand, when the formula (10) is established, EdRG indicates that the image captured by the image sensor 13 is in the rear pin state with respect to the edge E1 of the black and white chart S1. Also, the difference between EdRG and the threshold value “1” indicates the defocus amount of the edge E1 of the black and white chart S1 in the image captured by the image sensor 13.

  As described above, in the above formulas (7) to (10), an image captured by the image sensor 13 using the ratio between the red (R) color component amount and the green (G) color component amount is obtained. It is determined which of the front pin state and the rear pin state is in relation to the edge E1 of the black and white chart S1. However, using the difference between the red (R) color component amount and the green (G) color component amount, the image picked up by the image pickup device 13 is in the front pin state and the rear pin with respect to the edge E1 of the black and white chart S1. You may determine about which state among states.

  Similarly, as shown in FIG. 11B, when the image captured by the image sensor 13 is in the front pin state with respect to the edge E1 of the black and white chart S1, the color component amount of blue (B) The gradient is larger than the gradient of the color component amount of green (G). On the other hand, when the image picked up by the image pickup device 13 is in the back pinned state with respect to the edge E1 of the black and white chart S1, the gradient of the blue (B) color component amount is the green (G) color component amount. It becomes smaller than the gradient. Therefore, depending on whether the image captured by the image sensor 13 is in the front pin state or the rear pin state, the gradient of the color component amount of blue (B) at the edge E1 of the black and white chart S1 and the green ( The magnitude relationship with the gradient of the color component amount of G) is reversed.

  Therefore, by comparing Ed (hereinafter referred to as “EdBG”) based on the blue (B) color component amount and the green (G) color component amount with a threshold value, an image captured by the image sensor 13 is a black and white chart. Whether the state is the front pin state or the rear pin state with respect to the edge E1 of S1 is determined based on the following expressions (11) and (12).

  Here, Σ (B / G) is a sum obtained by adding a value obtained by dividing the blue (B) color component amount by the green (G) color component amount at the same position in the image over the section Δ3. Show. Then, a value obtained by dividing the sum by the length of the section Δ3 is calculated as EdBG. The section Δ3 is the green (G) color in the section located on the right side of the intersection P2 where the blue (B) color component amount and the green (G) color component amount intersect in FIG. It is defined as a part of a section where the component amount has a gradient.

  And when Formula (11) is materialized, EdBG has shown that the image imaged by the image pick-up element 13 is a front pin state with respect to the edge E1 of the black-and-white chart S1. On the other hand, when the formula (12) is established, EdBG indicates that the image captured by the image sensor 13 is in a rear pin state with respect to the edge E1 of the black and white chart S1. The difference between EdBG and the threshold value “1” indicates the defocus amount of the edge E1 of the black-and-white chart S1 in the image captured by the image sensor 13.

  It should be noted that whether the image captured by the image sensor 13 is in the front pin state or the rear pin state with respect to the edge E1 of the black and white chart S1 is expressed by the following equations (13) and (14). It may be determined based on.

  Here, Σ (B / G) is a sum obtained by adding the value obtained by dividing the blue (B) color component amount by the green (G) color component amount at the same position in the image over the section Δ4. Show. Then, a value obtained by dividing the sum by the length of the section Δ4 is calculated as EdBG. The section Δ4 is the green (G) color in the section located on the left side of the intersection P2 where the blue (B) color component amount and the green (G) color component amount intersect in FIG. It is defined as a part of a section where the component amount has a gradient.

  And when Formula (13) is materialized, EdBG has shown that the image imaged by the image pick-up element 13 is a back pin state with respect to the edge E1 of the black-and-white chart S1. On the other hand, when the formula (14) is established, EdBG indicates that the image captured by the image sensor 13 is in the front pin state with respect to the edge E1 of the black and white chart S1. The difference between EdBG and the threshold value “1” indicates the defocus amount of the edge E1 of the black-and-white chart S1 in the image captured by the image sensor 13.

  As described above, in the above formulas (11) to (14), an image captured by the image sensor 13 using the ratio between the color component amount of blue (B) and the color component amount of green (G) is obtained. It is determined which of the front pin state and the rear pin state is in relation to the edge E1 of the black and white chart S1. However, using the difference between the color component amount of blue (B) and the color component amount of green (G), the image picked up by the image sensor 13 is in the front pin state and the back pin with respect to the edge E1 of the black and white chart S1. You may determine about which state among states.

Next, an outline of the principle of calculating the direction index and the defocus amount in the subject using Wd will be described.
12A and 12B show the difference between the color component amounts of red (R) and green (G) in the vicinity of the edge E1 of the black and white chart S1, and the X axis in the detection target area EA of the edge E1. It is a graph which shows the relationship with the position in a direction.

  FIG. 12A shows a state in which the defocus amount of the edge E1 of the black and white chart S1 in the image picked up by the image pickup device 13 is small when the image pickup device 13 of the comparative example is used. Also, in FIG. 12A, a graph when the image captured by the image sensor 13 is in a back-pin state with respect to the edge E1 of the black-and-white chart S1 is indicated by a solid line, and the image captured by the image sensor 13 A graph in the case where the image is in the front pin state with respect to the edge E1 of the black and white chart S1 is indicated by a broken line. As shown in FIG. 12A, red (R) depending on whether the image captured by the image sensor 13 is in the front pin state or the rear pin state with respect to the edge E1 of the black and white chart S1. ) And green (G), the polarity (positive / negative) of the waveform for the difference in color component amount is reversed. However, in the state shown in FIG. 12A, the difference between the color component amounts of red (R) and green (G) in the vicinity of the edge E1 of the black and white chart S1 is small. Therefore, the waveform regarding the difference between the color component amounts of red (R) and green (G) is easily affected by an error that occurs when detecting the color component amounts of red (R) and green (G). Therefore, based on the polarity (positive / negative) of the waveform regarding the difference between the color component amounts of red (R) and green (G), the image picked up by the image pickup device 13 is in the front pin state with respect to the edge E1 of the black and white chart S1. And it is difficult to accurately determine which of the rear pin states.

  On the other hand, FIG. 12B shows a state where the defocus amount of the edge E1 of the black and white chart S1 in the image captured by the image sensor 13 is small when the image sensor 13 of the present embodiment is used. ing. In FIG. 12B, a graph in the case where the image captured by the image sensor 13 is in a back-pin state with respect to the edge E1 of the black and white chart S1 is indicated by a solid line, and the image captured by the image sensor 13 is illustrated. A graph in the case of the front pin state with respect to the edge E1 of the black and white chart S1 is indicated by a broken line. In the state shown in FIG. 12B, red (R) and green (in the vicinity of the edge E1 of the black-and-white chart S1 are compared with the case where the imaging element 13 of the comparative example shown in FIG. The difference in the color component amount of G) is large. Therefore, based on the polarity (positive / negative) of the waveform with respect to the difference between the color component amounts of red (R) and green (G), the image picked up by the image pickup device 13 is in the front pin state with respect to the edge E1 of the black and white chart S1. It is possible to determine which of the rear pin states.

  Further, as shown in FIG. 12B, Wd is defined as the distance between peaks of the waveform of the difference between the color component amounts of red (R) and green (G), and is expressed by the following equation (15). expressed.

  Here, X1 is a position where the difference between the color component amounts of red (R) and green (G) is minimized in the graph indicated by the solid line in FIG. On the other hand, X2 is a position where the difference between the color component amounts of red (R) and green (G) is maximized in the graph indicated by the solid line in FIG. The value of Wd indicates the defocus amount of the edge E1 of the black and white chart S1 in the image picked up by the image pickup device 13.

  Note that, using the ratio of the color component amounts of red (R) and green (G), the image captured by the image sensor 13 is in any of the front pin state and the rear pin state with respect to the edge E1 of the black and white chart S1. It may be determined whether it is in a state, or the defocus amount of the edge E1 of the black and white chart S1 may be calculated. In addition, the combination of color components to be compared is not limited to the combination of red (R) and green (G), but the combination of red (R) and blue (B), green (G) and blue (B). A combination of these may be used.

Next, an outline of the principle of calculating the direction index and the defocus amount in the subject using the LSF will be described.
LSF is a type of function that represents the spread of color component amounts in an image. FIG. 13A shows the LSF in which the image captured by the image sensor 13 is in focus at the edge E1 of the black and white chart S1. FIG. 13B shows an LSF in which an image captured by the image sensor 13 is out of focus at the edge E1 of the black and white chart S1. FIG. 13C illustrates an LSF in a state where the defocus amount of the edge E1 of the black and white chart S1 in the image captured by the image sensor 13 is larger than the state illustrated in FIG.

  As shown in FIGS. 13A to 13C, the standard deviation σ of the LSF at the edge E1 of the black and white chart S1 has a large defocus amount of the edge E1 of the black and white chart S1 in the image picked up by the image sensor 13. As it gets, it grows gradually.

  FIG. 14A shows the LSF for each color component when the image captured by the image sensor 13 is in focus at the edge E1 of the black and white chart S1. As shown in FIG. 14A, in the state where the image picked up by the image pickup device 13 is focused on the edge E1 of the black and white chart S1, the LSFs in all the color components have a sharp waveform. The standard deviations σR, σG, and σB of LSF in these color components are almost equal to each other.

  FIG. 14B shows the LSF for each color component when the image picked up by the image pickup device 13 is in the front pin state with respect to the edge E1 of the black and white chart S1. In FIG. 14B, in the case where the image pickup device 13 of the comparative example is used, the LSF in a state where the defocus amount of the edge E1 of the black and white chart S1 in the image picked up by the image pickup device 13 is very small is indicated by a broken line. Has been. In the state indicated by the broken line in FIG. 14B, since the difference in the defocus amount of the edge E1 of the black and white chart S1 in which the light that has passed through the focus lens 11 is captured by the image sensor 13 for each color component is small, all the color components The LSFs have substantially the same waveform. Therefore, the standard deviations σR, σG, and σB of LSF in these color components are almost equal to each other.

  FIG. 14B shows the LSF in a state where the defocus amount of the edge E1 of the black and white chart S1 in the image picked up by the image pickup device 13 is small when the image pickup device 13 of the present embodiment is used. It is shown with a solid line. In the state shown by the solid line in FIG. 14B, even if the defocus amount of the edge E1 of the black and white chart S1 in the image picked up by the image pickup device 13 is small, the light passing through the focus lens 11 is separated for each color component. In addition, the difference in the defocus amount of the edge E1 of the black and white chart S1 imaged on the image sensor 13 is larger than the state indicated by the broken line in FIG. Therefore, the standard deviations σR, σG, and σB of the LSF of each color component are different from each other, and gradually increase in the order of blue (B), green (G), and red (R). This is because when the image captured by the image sensor 13 is in the front pin state with respect to the edge E1 of the black and white chart S1, the black and white chart in which the light passing through the focus lens 11 is captured by the image sensor 13 for each color component. This is because the defocus amount of the edge E1 of S1 gradually increases in the order of blue (B) light, green (G) light, and red (R) light.

  FIG. 14C shows the LSF for each color component when the image picked up by the image pickup device 13 is in a back-pin state with respect to the edge E1 of the black and white chart S1. In FIG. 14C, when the image pickup device 13 of the comparative example is used, the LSF in a state where the defocus amount of the edge E1 of the black and white chart S1 in the image picked up by the image pickup device 13 is very small is indicated by a broken line. Has been. In the state indicated by the broken line in FIG. 14C, since the difference in the defocus amount of the edge E1 of the black and white chart S1 in which the light passing through the focus lens 11 is imaged on the image sensor 13 for each color component is small, all the color components The LSFs have substantially the same waveform. Therefore, the standard deviations σR, σG, and σB of LSF in these color components are almost equal to each other.

  FIG. 14C shows the LSF in a state where the defocus amount of the edge E1 of the black and white chart S1 in the image captured by the image sensor 13 is small when the image sensor 13 of the present embodiment is used. It is shown with a solid line. In the state indicated by the solid line in FIG. 14C, even if the defocus amount of the edge E1 of the black and white chart S1 in the image picked up by the image pickup device 13 is small, the light passing through the focus lens 11 is separated for each color component. In addition, the difference in the defocus amount of the edge E1 of the black and white chart S1 imaged on the image sensor 13 is larger than the state indicated by the broken line in FIG. Therefore, the standard deviations σR, σG, and σB of the LSF of each color component are different from each other, and gradually increase in the order of red (R), green (G), and blue (B). This is because the image captured by the image sensor 13 is in a back-pin state with respect to the edge E1 of the black and white chart S1, and the black and white chart in which the light passing through the focus lens 11 is captured by the image sensor 13 for each color component. This is because the defocus amount of the edge E1 of S1 gradually increases in the order of red (R) light, green (G) light, and blue (B) light.

  That is, in the state indicated by the solid line in FIGS. 14B and 14C, the standard deviation σR of the LSF of each color component is compared with the state indicated by the broken line in FIGS. 14B and 14C. The difference between σG and σB is large. The LSF of each color component at the edge E1 of the black and white chart S1 depends on whether the image picked up by the image sensor 13 is in the front pin state or the rear pin state with respect to the edge E1 of the black and white chart S1. The magnitude relations of the standard deviations are different from each other.

  FIG. 15 is a graph showing the relationship between the distance of the edge E1 of the black and white chart S1 in the image picked up by the image pickup device 13 and the difference in standard deviation of the LSF of each color at the edge E1. In FIG. 15, a graph when the image sensor 13 of the comparative example is used is indicated by a broken line, and a graph when the image sensor 13 of the present embodiment is used is indicated by a solid line.

  Here, F1 is a function representing the difference between the standard deviation σB of the blue (B) LSF and the standard deviation σG of the green (G) LSF. F1 is a positive value when the image picked up by the image pickup device 13 is in a rear pin state with respect to the edge E1 of the black and white chart S1. On the other hand, F1 takes a negative value when the image picked up by the image pickup device 13 is in the front pin state with respect to the edge E1 of the black and white chart S1. Further, the absolute value of F1 gradually increases as the distance of the edge E1 of the black and white chart S1 in the image picked up by the image pickup device 13 moves away from the distance corresponding to the in-focus position. Therefore, the absolute value of F1 indicates the defocus amount of the edge E1 of the black and white chart S1 in the image captured by the image sensor 13.

  Note that when the image sensor 13 of the present embodiment is used, the difference between the standard deviations σR, σG, and σB of the LSF of each color component is larger than when the image sensor 13 of the comparative example is used. Therefore, the F1 function graph indicated by the solid line in FIG. 15 has a larger absolute value of F1 than the F1 function graph indicated by the broken line in FIG. 15, and therefore, the blue (B) and green (G) LSF standard Less susceptible to errors that occur when calculating deviations. Therefore, even if the defocus amount of the edge E1 of the black-and-white chart S1 in the image picked up by the image pickup device 13 is small, the image pickup device 13 picks up the image based on the graph of the function F1 indicated by the solid line in FIG. It is possible to accurately determine whether the image is in the front pin state or the rear pin state with respect to the edge E1 of the black and white chart S1. Further, even if the defocus amount of the edge E1 of the black and white chart S1 in the image captured by the image sensor 13 is small, the image is captured by the image sensor 13 based on the function graph of F1 indicated by the solid line in FIG. It is possible to accurately calculate the defocus amount of the edge E1 of the black and white chart S1 in the obtained image.

  Similarly, F2 is a function representing the difference between the standard deviation σR of the red (R) LSF and the standard deviation σG of the green (G) LSF. F2 is a negative value when the image picked up by the image pickup device 13 is in a rear pin state with respect to the edge E1 of the black and white chart S1. On the other hand, F2 is a positive value when the image captured by the image sensor 13 is in the front pin state with respect to the edge E1 of the black and white chart S1. In addition, the absolute value of F2 gradually increases as the distance of the edge E1 of the black and white chart S1 in the image captured by the image sensor 13 increases from the distance corresponding to the in-focus position. Therefore, the absolute value of F1 indicates the defocus amount of the edge E1 of the black and white chart S1 in the image captured by the image sensor 13.

  Note that when the image sensor 13 of the present embodiment is used, the difference between the standard deviations σR, σG, and σB of the LSF of each color component is larger than when the image sensor 13 of the comparative example is used. Therefore, the F2 function graph shown by the solid line in FIG. 15 has a larger absolute value of F1 than the F2 function graph shown by the broken line in FIG. 15, and therefore the red (R) and green (G) LSF standard Less susceptible to errors that occur when calculating deviations. Therefore, even if the defocus amount of the edge E1 of the black and white chart S1 in the image captured by the image sensor 13 is small, the image is captured by the image sensor 13 based on the graph of the function F2 indicated by the solid line in FIG. It is possible to accurately determine whether the image is in the front pin state or the rear pin state with respect to the edge E1 of the black and white chart S1. Further, even if the defocus amount of the edge E1 of the black and white chart S1 in the image captured by the image sensor 13 is small, the image is captured by the image sensor 13 based on the function graph of F2 indicated by the solid line in FIG. It is possible to accurately calculate the defocus amount of the edge E1 of the black and white chart S1 in the obtained image.

  As described above, based on the polarity of the difference in the standard deviation of the LSF of the color component amounts of red (R) and green (G) or blue (B) and green (G) at the edge E1, the image sensor 13 It is determined whether the captured image is in the front pin state or the rear pin state with respect to the edge E1 of the black and white chart S1. However, based on the polarity of the difference between the standard deviations of the red (R) and blue (B) LSF at the edge E1, the image picked up by the image sensor 13 is in the front pin state and the rear with respect to the edge E1 of the black and white chart S1. It may be determined which of the pin states. Further, the ratio of the standard deviation of the LSF of each color at the edge E1 is compared with a threshold value (= “1”), whereby the image captured by the image sensor 13 is in the front pin state and the rear with respect to the edge E1 of the black and white chart S1. It may be determined which of the pin states.

  The LSF may be calculated for only one color component among red (R), green (G), and blue (B). In this case, as an example, first, an edge profile that associates the LSF value at the edge E1 with the defocus amount of the edge E1 is stored in advance in the database. Then, by comparing the LSF value at the edge E1 of the black and white chart S1 imaged by the image sensor 13 with the profile of the edge stored in the database, the edge E1 of the black and white chart S1 in the image captured by the image sensor 13 is checked. A defocus amount is calculated.

Further, when calculating the direction index and the defocus amount in the subject, the half-value width of the LSF or the peak value of the LSF may be used instead of the standard deviation of the LSF.
Next, an outline of an image generation processing routine executed by the MPU 16 of the image processing engine 14 of the present embodiment will be described with reference to the flowchart of FIG.

  Now, when the camera 10 is turned on, the MPU 16 starts an image generation processing routine shown in FIG. In step S10, the MPU 16 reads an image signal from the image sensor 13 and generates a through image.

  Specifically, as illustrated in FIG. 17A, the MPU 16 generates a frame image F based on pixel signals read from the flat pixels, the convex pixels 40RA, and the concave pixels 40BA that are all imaging pixels of the imaging element 13. Generate a through image by connecting the generated frame images F in chronological order. Then, the display control unit 25 of the MPU 16 displays the generated through image on the monitor 21 as an image captured by the image sensor 13.

  In the example shown in FIG. 17A, the frame rate of the frame image F is set to “30”. Here, the frame rate means the number of frame images included per unit time in a through image. That is, the frame rate is also an index indicating the length of time for each frame image F in the through image, and the length of time for each frame image F in the through image decreases as the frame rate increases.

  Also, as shown in FIG. 17B, the MPU 16 includes a first frame image F1 generated based on the pixel signal read from the flat pixels of the image sensor 13, the flat pixels of the image sensor 13, the convex pixels 40RA, and A through image may be generated by connecting the second frame image F2 generated based on the pixel signal read from the concave pixel 40BA in time series. In this case, the MPU 16 may generate the first frame image F1 by selectively reading out the pixel signal from the flat pixels by the thinning readout process among all the imaging pixels 40 of the imaging element 13. Further, although the MPU 16 reads pixel signals from all the imaging pixels 40 of the imaging element 13, the MPU 16 may generate the first frame image F1 using only the pixel signals read from the flat pixels among the read pixel signals. In the example shown in FIG. 17B, the frame rates of both the first frame image F1 and the second frame image F2 are set to “60”.

  Then, in step S11, the edge detection unit 22 of the MPU 16 scans the through image generated in the previous step S10 with a differential filter (for example, raster scan). As a result, feature quantities such as lightness, saturation, and hue in the through image are calculated. Then, the edge detection unit 22 of the MPU 16 detects a portion having a large calculated feature amount as an edge suitable for evaluation of axial chromatic aberration.

  Subsequently, in step S12, the focus detection unit 23 of the MPU 16 calculates the evaluation value of the axial chromatic aberration at the edge detected in the previous step S11, and based on the calculated evaluation value, the defocus feature value of the edge Is calculated.

More specifically, in step S11 and step S12 described above, the MPU 16 performs the following process.
That is, when the through image is the example shown in FIG. 17A, the edge detection unit 22 of the MPU 16 is generated based on the pixel signals read from the flat pixels, the convex pixels 40RA, and the concave pixels 40BA of the image sensor 13. Edges are detected by scanning the frame image F with a differential filter. Next, the MPU 16, for example, among the edges detected from the frame image F, the red (R) color component amount of the image portion captured by the red pixel 40R that is the convex pixel 40RA and the green pixel that is a flat pixel. Based on the green (G) color component amount of the image portion captured by 40G, a normalized output of the color component amount in the vicinity of the edge is calculated.

  The MPU 16 also detects the blue (B) color component amount of the image portion captured by the blue pixel 40B that is the concave pixel 40BA and the image captured by the green pixel 40G that is a flat pixel among the edges detected from the frame image F. Based on the green (G) color component amount of the part, the normalized output of the color component amount in the vicinity of the edge may be calculated.

  When the through image is the example shown in FIG. 17B, the edge detection unit 22 of the MPU 16 is generated based on the pixel signals read from the flat pixels, the convex pixels 40RA, and the concave pixels 40BA of the image sensor 13. Edges are detected by scanning the second frame image F2 with a differential filter. Next, as an example, the MPU 16 is a flat pixel and the red (R) color component amount of the image portion captured by the red pixel 40R which is the convex pixel 40RA among the edges detected from the second frame image F2. Based on the green (G) color component amount of the image portion captured by the green pixel 40G, a normalized output of the color component amount in the vicinity of the edge is calculated.

  Also, the MPU 16 captures the blue (B) color component amount of the image portion captured by the blue pixel 40B that is the concave pixel 40BA and the green pixel 40G that is the flat pixel among the edges detected from the second frame image F2. The normalized output of the color component amount in the vicinity of the edge may be calculated based on the green (G) color component amount of the image portion.

  Next, in step S13, the MPU 16 sets the drive condition for the focus lens 11 based on the edge defocus feature value calculated in the previous step S12. That is, the MPU 16 sets the drive direction of the focus lens 11 based on the direction index of the edge defocus feature value and sets the drive amount of the focus lens 11 based on the defocus value of the edge defocus feature value. .

  Specifically, the MPU 16 sets the drive direction of the focus lens 11 in a direction away from the image sensor 13 in the optical axis direction when the direction index of the edge defocus feature value indicates a front pin state. To do. In other words, when the MPU 16 determines that the through image is in focus closer to the near side than the edge, the MPU 16 moves the focus position of the through image to the infinity side in the depth direction of the through image to change the position of the edge. The driving direction of the focus lens 11 is set so as to match.

  On the other hand, the MPU 16 sets the drive direction of the focus lens 11 in the direction in which the focus lens 11 is brought closer to the image sensor 13 in the optical axis direction when the direction index of the edge defocus feature value indicates the rear pin state. That is, when the MPU 16 determines that the through image is in focus on the far point side of the edge, the MPU 16 moves the focus position of the through image to the closest side in the depth direction of the through image to the edge position. The drive direction of the focus lens 11 is set so as to match.

  Further, the MPU 16 estimates the in-focus position of the focus lens 11 based on the defocus amount of the edge defocus feature amount. Then, the MPU 16 sets the drive amount of the focus lens 11 so as to reach the in-focus position estimated from the current lens position.

  Subsequently, in step S14, the MPU 16 drives the focus lens 11 under the driving conditions set in the previous step S13. As a result, when the lens position of the focus lens 11 is moved to the in-focus position, the through image is brought into focus at the edge position.

  Next, in step S15, the MPU 16 determines whether or not a shooting instruction signal has been input. This photographing instruction signal is input as an operation signal to the image processing engine 14 when the release button 31 is fully pressed. When the monitor 21 is a touch panel, a shooting instruction signal is input to the image processing engine 14 as an operation signal when the monitor 21 is touched for shooting an image. If the MPU 16 determines that the shooting instruction signal is not input (step S15 = NO), the MPU 16 returns the process to step S10, and repeats the processes of steps S10 to S15 until the shooting instruction signal is input. On the other hand, if the MPU 16 determines that a shooting instruction signal has been input (step S15 = YES), the MPU 16 proceeds to step S16.

  In step S16, the MPU 16 is a graph GR indicating the correlation between the shading coefficient and the aperture value in the concave pixel 40BA, which is a criterion for determining whether or not to use the pixel signal of the subject captured by the concave pixel 40BA in the captured image. (See FIG. 18) is read from the nonvolatile memory 18.

  The shading coefficient in the concave pixel 40BA is calculated by integrating the shading coefficient of the focus lens 11, the pixel sensitivity of the concave pixel 40BA, and the concave coefficient of the concave pixel 40BA. In this case, the dent coefficient of the concave pixel 40BA increases as the dent degree of the concave pixel 40BA increases. Therefore, the shading coefficient in the concave pixel 40BA increases as the concave degree of the concave pixel 40BA increases. That is, the shading coefficient in the concave pixel 40BA is an index indicating the degree of dent in the concave pixel 40BA.

  In the graph GR described above, as the shading coefficient in the concave pixel 40BA increases, the size of the aperture value that serves as a criterion for determining whether or not to use the image signal of the subject imaged by the concave pixel 40BA in the captured image. Is gradually growing. This is because the aperture area of the diaphragm is reduced as the degree of depression of the concave pixel 40BA increases in order to ensure a sufficient ratio of the light that has passed through the focus lens 11 and reaches the concave pixel 40BA. This is because it is necessary to limit the incident angle of light.

Subsequently, in step S17, the MPU 16 calculates the setting value of the shading coefficient in the concave pixel 40BA set in the camera 10 at that time.
Specifically, when the camera 10 is a lens-integrated type, the shading coefficient of the focus lens 11 attached to the camera 10 has a fixed value. Therefore, by setting the pixel sensitivity of the concave pixel 40BA and the concave coefficient of the concave pixel 40BA to the shading coefficient of the focus lens 11 that is a fixed value, the setting value of the shading coefficient in the concave pixel 40BA is calculated. The

  Further, when the camera 10 is an interchangeable lens type, the shading coefficient of the focus lens 11 varies depending on the type of focus lens attached to the camera 10. Therefore, the shading coefficient of the focus lens 11 corresponding to the type of the focus lens 11 attached to the camera 10 at that time is read from the nonvolatile memory 18. Then, by adding the pixel sensitivity of the concave pixel 40BA and the concave coefficient of the concave pixel 40BA to the read shading coefficient of the focus lens 11, the setting value of the shading coefficient in the concave pixel 40BA is calculated. .

  Next, in step S18, the MPU 16 calculates the threshold value N of the aperture value by adding the setting value of the shading coefficient calculated in the previous step S17 to the graph GR read in the previous step S16. The threshold value N is a reference value for the aperture value when determining whether or not to use the image signal of the subject imaged by the concave pixel 40BA for the captured image. FIG. 18 shows the threshold value N of the aperture value calculated when the setting value of the shading coefficient is T1.

  Subsequently, in step S19, the MPU 16 determines whether or not the aperture value set in the camera 10 at that time is equal to or greater than the threshold value threshold N calculated in the previous step S18. In the example illustrated in FIG. 18, when the aperture value setting value is N1, the MPU 16 determines that the aperture value setting value is less than the threshold value N (step S19 = NO), while the aperture value is Is set to N2, it is determined that the aperture value setting value is greater than or equal to the threshold value N (step S19 = YES). When the MPU 16 determines that the aperture value setting value is equal to or greater than the threshold value N (step S19 = YES), it is appropriate to use the image signal of the subject captured by the concave pixel 40BA for the captured image. After the determination, the process proceeds to step S20.

  In step S20, the image generation unit 24 of the MPU 16 reads the pixel signal of the subject from all the imaging pixels 40 (that is, the flat pixel, the convex pixel 40RA, and the concave pixel 40BA) including the concave pixel 40BA.

  Next, in step S21, the image generation unit 24 of the MPU 16 performs normal processing on the pixel signal of the subject read out from the imaging pixel 40 (that is, the flat pixel and the convex pixel 40RA) other than the concave pixel 40BA in the previous step S20. The shading correction process using the shading coefficient is executed.

  Subsequently, in step S22, the image generation unit 24 of the MPU 16 sets the shading coefficient setting value in the concave pixel 40BA calculated in the previous step S17 with respect to the pixel signal of the subject read from the concave pixel 40BA in the previous step S20. The shading correction process using is executed. In this case, the setting value of the shading coefficient in the concave pixel 40BA is larger than the normal shading coefficient used in the previous step S21. This is because light obliquely incident on the image sensor 13 out of light that has passed through the focus lens 11 is less likely to reach the concave pixel 40BA than the flat pixel, and the pixel signal of the subject read from the concave pixel 40BA. This is because it is necessary to increase the strength of the shading correction process.

  Next, in step S23, the image generation unit 24 of the MPU 16 performs magnification correction processing on the subject pixel signal read from the convex pixel 40RA or the concave pixel 40BA in the previous step S20, and then performs the processing. Control goes to step S28.

  Specifically, the image generation unit 24 of the MPU 16 performs an enlargement process on the pixel signal of the subject read from the convex pixel 40RA. This is because the convex pixel 40RA has a smaller distance between the photoelectric conversion unit 43RA and the focus lens 11 in the optical axis direction of the incident light than the flat pixel, and thus the size of the subject imaged by the convex pixel 40RA is flat. This is because it is smaller than the size of the same subject imaged on the pixel.

  In addition, the image generation unit 24 of the MPU 16 performs a reduction process on the pixel signal of the subject read from the concave pixel 40BA. This is because the concave pixel 40BA has a larger distance between the photoelectric conversion unit 43BA and the focus lens 11 in the optical axis direction of the incident light than the flat pixel, and thus the size of the subject imaged on the concave pixel 40BA is flat. This is because it is larger than the size of the same subject imaged on the pixel.

  On the other hand, when the MPU 16 determines that the set value of the aperture value is less than the threshold value N (step S19 = NO), it is not appropriate to use the image data of the subject captured by the concave pixel 40BA for the captured image. After the determination, the process proceeds to step S24.

  In step S24, the image generation unit 24 of the MPU 16 selectively reads out pixel signals of the subject from the imaging pixels 40 (that is, the flat pixels and the convex pixels 40RA) other than the concave pixels 40BA by the thinning-out reading process.

  The image generation unit 24 of the MPU 16 may read the subject pixel signal from all the imaging pixels 40 including the concave pixel 40BA, but may not use the subject pixel signal read from the concave pixel 40BA in the captured image. .

  Next, in step S25, the image generation unit 24 of the MPU 16 executes a shading correction process using a normal shading coefficient on the subject pixel signal read from the flat pixel and the convex pixel 40RA in the previous step S24. .

  Subsequently, in step S26, the image generation unit 24 of the MPU 16 performs an enlargement process as a magnification correction process on the pixel signal of the subject read from the convex pixel 40RA in the previous step S24.

  Next, in step S27, the image generation unit 24 of the MPU 16 performs interpolation processing on the pixel signal of the subject read from the flat pixel and the convex pixel 40RA in the previous step S24, thereby corresponding to the concave pixel 40BA. After supplementing the pixel signals for the imaging pixels, the process proceeds to step S28.

  In step S28, the image generation unit 24 of the MPU 16 generates a still image based on the pixel signal subjected to the magnification correction process in the previous step S23 or the pixel signal subjected to the interpolation process in the previous step S27. The generated still image is stored in the nonvolatile memory 18 as a captured image.

  Next, the operation of the camera 10 configured as described above will be described below, particularly focusing on the operation when the focus detection unit 23 of the MPU 16 calculates the edge defocus feature amount in the through image.

  Now, it is assumed that the red pixel 40R that is the convex pixel 40RA and the green pixel 40G that is a flat pixel have captured the edges included in the through image. In this case, the normalized output of the red (R) color component amount in the vicinity of the edge imaged by the red pixel 40R, which is the convex pixel 40RA, and the green (G) in the vicinity of the edge imaged by the green pixel 40G, which is a flat pixel. The difference between the color component amount and the normalized output is larger than when the red pixel 40R is a flat pixel. That is, the difference between the normalized output of the red (R) and green (G) color component amounts in the vicinity of the edge is expanded by the combination of the red pixel 40R as the convex pixel 40RA and the green pixel 40G as the flat pixel. . Therefore, even when the defocus amount of the edge included in the through image is small, a sufficient difference between the normalized output of the red (R) and green (G) color component amounts in the vicinity of the edge is ensured. The Then, an evaluation value of axial chromatic aberration at the edge is calculated based on the difference between the enlarged normalized outputs, and an edge defocus feature value is calculated based on the calculated evaluation value. Therefore, the driving condition of the focus lens 11 is set according to the calculation result of the edge defocus feature amount in the through image, so that the through image is accurately focused on the edge portion.

  Similarly, it is assumed that the blue pixel 40B, which is the concave pixel 40BA, and the green pixel 40G, which is the flat pixel, capture an edge included in the through image. In this case, the normalized output of the blue (B) color component amount in the vicinity of the edge imaged by the blue pixel 40B as the concave pixel 40BA and the green (G) in the vicinity of the edge imaged by the green pixel 40G as the flat pixel. The difference between the color component amount and the normalized output is larger than when the blue pixel 40B is a flat pixel. That is, by combining the blue pixel 40B, which is the blue pixel 40B, and the green pixel 40G, which is a flat pixel, the difference between the normalized output of the blue (B) and green (G) color component amounts in the vicinity of the edge is expanded. . Therefore, even when the defocus amount of the edge included in the through image is small, a sufficient difference between the normalized output of the blue (B) and green (G) color component amounts in the vicinity of the edge is ensured. The Then, an evaluation value of axial chromatic aberration at the edge is calculated based on the difference between the enlarged normalized outputs, and an edge defocus feature value is calculated based on the calculated evaluation value. Therefore, the driving condition of the focus lens 11 is set according to the calculation result of the edge defocus feature amount in the through image, so that the through image is accurately focused on the edge portion.

  In particular, in the present embodiment, at least some of the frame images included in the through image include an image portion captured by the red pixel 40R that is the convex pixel 40RA or the blue pixel 40B that is the concave pixel 40BA, and a flat pixel. And the image portion captured by the green pixel 40G. Therefore, the difference between the normalized output of the red (R) and green (G) color component amounts in the vicinity of the edge detected from the frame image, or the normal of the blue (B) and green (G) color component amounts. The evaluation value of the axial chromatic aberration of the edge in the through image is calculated as needed in the state where the difference in the digitized output is enlarged. Accordingly, the defocus feature amount of the edge in the through image is calculated as needed based on the evaluation value, and the driving condition of the focus lens 11 is updated according to the calculation result of the defocus feature amount. It becomes possible to continuously focus accurately in the portion.

According to the first embodiment, the following effects can be obtained.
(1) Even if the defocus amount is small and the edge is not suitable for the detection of the in-focus state, the difference in the color component amount between the color components at such an edge is enlarged, so that the color of the edge A sufficient difference in color component amount or ratio between components is ensured. Then, based on the difference or ratio of the color component amounts between the edge color components, the in-focus state of the edge can be accurately detected.

  (2) A photoelectric conversion surface of an imaging pixel on which incident light of a color component on a relatively long wavelength side is incident, and a photoelectric conversion surface of an imaging pixel on which incident light of a color component on a relatively short wavelength side is incident By varying the position of the incident light in the optical axis direction, it is possible to easily realize a configuration in which the difference in the color component amount between the color components of the edges to be compared is detected when the in-focus state is detected. .

  (3) The imaging pixel 40 of the imaging device 13 has incident light compared to the other red pixels 40R among the red pixels 40R into which the incident light of red (R), which is a relatively long wavelength side color component, is incident. The convex pixel 40RA is disposed so as to protrude toward the focus lens 11 in the optical axis direction. In addition, the imaging pixel 40 of the imaging device 13 is configured to receive incident light as compared with the other blue pixels 40B among the blue pixels 40B on which incident light of blue (B), which is a relatively short wavelength side color component, is incident. It includes a concave pixel 40BA that is recessed in the direction opposite to the focus lens 11 in the optical axis direction. Therefore, the photoelectric conversion surface of the imaging pixel on which the incident light of the color component on the relatively long wavelength side is incident is more than the photoelectric conversion surface of the imaging pixel on which the incident light of the color component on the relatively short wavelength side is incident. A configuration arranged on the focus lens 11 side in the optical axis direction of incident light can be easily realized.

  (4) A captured image is generated by appropriately using a pixel signal captured by at least one of the convex pixel 40RA and the concave pixel 40BA. Therefore, it is possible to secure a larger number of pixels of the captured image compared to the case where the captured image is generated without using the pixel signal captured by the convex pixel 40RA and the concave pixel 40BA.

  (5) A captured image is generated after image processing different from pixel signals captured by other imaging pixels in the image sensor 13 is performed on pixel signals captured by at least one of the convex pixels 40RA and the concave pixels 40BA. Is done. Therefore, the convex pixel 40RA and the concave pixel 40BA are imaged due to the difference in the position of the photoelectric conversion surface in the optical axis direction of incident light compared to the other imaging pixels in the imaging element. Even if an error occurs in the pixel signal, image processing for reducing the error is performed on the pixel signal captured by the convex pixel 40RA and the concave pixel 40BA. For this reason, even if a captured image is generated using a pixel signal captured by at least one of the convex pixel 40RA and the concave pixel 40BA, it is possible to suppress degradation of the image quality of the captured image.

  (6) The amount of incident light incident on the concave pixel 40BA is reduced because the concave pixel 40BA is recessed in the optical axis direction of the incident light as compared with the other imaging pixels in the image sensor 13. It can happen. In this regard, in the above embodiment, shading correction processing using a shading coefficient different from normal is performed on the pixel signal imaged by the concave pixel 40BA as image processing for reducing such a decrease in light amount. Therefore, even if a captured image is generated using a pixel signal imaged by the concave pixel 40BA, it is possible to suppress a decrease in the image quality of the captured image.

  (7) The convex pixels 40RA and the concave pixels 40BA are different from the other imaging pixels in the imaging device 13 because the positions of the photoelectric conversion surfaces in the optical axis direction of the incident light are different from each other. There may be a change in the magnification of the image picked up by. In this regard, in the above-described embodiment, the magnification correction process is performed on the pixel signals picked up by the convex pixel 40RA and the concave pixel 40BA as image processing for reducing such a change in magnification. For this reason, even when a captured image is generated using a pixel signal captured by at least one of the convex pixel 40RA and the concave pixel 40BA, it is possible to suppress degradation in the image quality of the captured image.

  (8) An image captured by an imaging pixel including at least one of the convex pixel 40RA and the concave pixel 40BA is displayed on the monitor 21 as a through image. Therefore, the image content of the through image in which the focus state of the edge is calculated based on the color component amount of the edge captured by at least one of the convex pixel 40RA and the concave pixel 40BA can be viewed through the monitor 21.

  (9) When a moving object is included in the through image, the in-focus state of the moving object is moved based on the color component amount of the edge of the moving object captured by at least one of the convex pixel 40RA and the concave pixel 40BA. It is possible to accurately calculate in real time while following the movement of the body.

(Second Embodiment)
Next, a second embodiment will be described. Note that the second embodiment differs from the first embodiment in that the focal lengths of the microlenses 41 corresponding to the light components of the respective colors out of the incident light incident on the image sensor 13 are different from each other. Therefore, in the following description, a configuration that is different from the first embodiment will be mainly described, and a configuration that is the same as or equivalent to that of the first embodiment will be denoted by the same reference numeral, and redundant description will be omitted.

  As shown in FIG. 19A, in the present embodiment, the imaging device 13 is configured such that a part of the microlens 41RA of the red pixel 40R constituting the imaging pixel 40 is compared with the microlens 41R of the other red pixel 40R. Thus, the curvature of the lens surface is small, and the focal length for red (R) incident light is long.

  Further, as shown in FIG. 19B, the imaging device 13 is configured such that a part of the microlenses 41BA of the blue pixel 40B constituting the imaging pixel 40 is compared with the microlenses 41B of the other blue pixels 40B. The curvature of the surface is large, and the focal length for blue (B) incident light is short.

  The imaging element 13 does not include the convex pixel 40RA as the red pixel 40R constituting the imaging pixel 40, and the photoelectric conversion surfaces of the photoelectric conversion units 43R of all the red pixels 40R are the same in the optical axis direction of the incident light. It is arranged at the position. Further, the imaging element 13 does not include the concave pixel 40BA as the blue pixel 40B constituting the imaging pixel 40, and the photoelectric conversion surfaces of the photoelectric conversion units 43B of all the blue pixels 40B are the same in the optical axis direction of the incident light. It is arranged at the position. That is, in the imaging element 13, the photoelectric conversion surfaces of the photoelectric conversion units 43 of all the imaging pixels 40 are arranged at the same position in the optical axis direction of incident light.

  In FIG. 20A, when the image picked up by the image pickup device 13 is in the front pin state with respect to the subject, the incident light of each color component that has passed through the microlens 41 enters the photoelectric conversion unit 43. The state in which the subject is imaged is shown. As shown in FIG. 20A, when the image picked up by the image pickup device 13 is in the front pin state with respect to the subject, the light passing through the microlens 41 is picked up by the photoelectric conversion unit 43 for each color component. The defocus amount of the subject to be increased gradually in the order of blue (B) light, green (G) light, and red (R) light.

  In particular, in this embodiment, the curvature of the micro lens 41RA in a part of the red pixel 40R is smaller than the curvature of the micro lens 41R in the other red pixel 40R. Therefore, when the red (R) incident light passes through the microlens 41RA having a relatively small curvature, the photoelectric light of the red pixel 40R is compared with the case of passing through the microlens 41R having a relatively large curvature. The defocus amount of the subject imaged by the conversion unit 43R increases. As a result, the defocus amount of the subject imaged by the red (R) incident light on the photoelectric conversion unit 43R of the red pixel 40R, and the defocus amount of the subject imaged by the green (G) light on the photoelectric conversion unit 43G of the green pixel 40G. The difference from the focus amount is larger than when the red (R) incident light passes through the microlens 41R having a relatively large curvature. In this case, the normalized output of the red (R) color component amount in the vicinity of the edge imaged by the red pixel 40R having the micro lens 41RA having a relatively small curvature, and the green (in the vicinity of the edge imaged by the green pixel 40G) The difference between the color component amount of G) and the normalized output is larger than when the red pixel 40R has the microlens 41R having a relatively large curvature. Therefore, even when the defocus amount of the edge included in the through image is small, a sufficient difference between the normalized output of the red (R) and green (G) color component amounts in the vicinity of the edge is ensured. The

  Further, some of the microlenses 41BA of the blue pixel 40B have a larger curvature of the lens surface than the microlenses 41B of the other blue pixels 40B. For this reason, when the blue (B) incident light passes through the microlens 41BA having a relatively large curvature, the photoelectric light of the blue pixel 40B is compared with the case where it passes through the microlens 41B having a relatively small curvature. The defocus amount of the subject imaged by the conversion unit 43B is reduced. As a result, the defocus amount of the subject imaged by the blue (B) incident light on the photoelectric conversion unit 43B of the blue pixel 40B and the defocus amount of the subject imaged by the green (G) light on the photoelectric conversion unit 43G of the green pixel 40G. The difference from the focus amount is larger than when the blue (B) incident light passes through the microlens 41B having a relatively small curvature. In this case, the normalized output of the blue (B) color component amount in the vicinity of the edge imaged by the blue pixel 40B having the relatively large curvature microlens 41BA and the green (in the vicinity of the edge imaged by the green pixel 40G) The difference between the color component amount of G) and the normalized output is larger than when the blue pixel 40B has a microlens 41B having a relatively small curvature. Therefore, even when the defocus amount of the edge included in the through image is small, a sufficient difference between the normalized output of the blue (B) and green (G) color component amounts in the vicinity of the edge is ensured. The

  In FIG. 20B, when the image picked up by the image pickup device 13 is in a back-pin state with respect to the subject, the incident light of each color component that has passed through the microlens 41 enters the photoelectric conversion unit 43. A state in which the subject is incident and the subject is imaged is shown. As shown in FIG. 20B, when the image picked up by the image pickup device 13 is in a back-pin state with respect to the subject, the light passing through the microlens 41 is picked up by the photoelectric conversion unit 43 for each color component. The defocus amount of the subject to be increased gradually in the order of red (R) light, green (G) light, and blue (B) light.

  In particular, in the present embodiment, some of the microlenses 41RA of the red pixel 40R have a smaller curvature of the lens surface than the microlenses 41R of the other red pixels 40R. Therefore, when the red (R) incident light passes through the microlens 41RA having a relatively small curvature, the photoelectric light of the red pixel 40R is compared with the case of passing through the microlens 41R having a relatively large curvature. The defocus amount of the subject imaged by the conversion unit 43 is reduced. As a result, the defocus amount of the subject imaged by the red (R) incident light on the photoelectric conversion unit 43R of the red pixel 40R, and the defocus amount of the subject imaged by the green (G) light on the photoelectric conversion unit 43G of the green pixel 40G. The difference from the focus amount is larger than when the red (R) incident light passes through the microlens 41R having a relatively large curvature.

  Further, some of the microlenses 41BA of the blue pixel 40B have a larger curvature of the lens surface than the microlenses 41B of the other blue pixels 40B. For this reason, when the blue (B) incident light passes through the microlens 41BA having a relatively large curvature, the photoelectric light of the blue pixel 40B is compared with the case where it passes through the microlens 41B having a relatively small curvature. The defocus amount of the subject imaged by the conversion unit 43B increases. As a result, the defocus amount of the subject imaged by the blue (B) incident light on the photoelectric conversion unit 43B of the blue pixel 40B and the defocus amount of the subject imaged by the green (G) light on the photoelectric conversion unit 43G of the green pixel 40G. The difference from the focus amount is larger than when the blue (B) incident light passes through the microlens 41B having a relatively small curvature.

  Therefore, even when the defocus amount of the edge included in the through image is small, red (R) and green (G) or blue (B) and green (G) color components in the vicinity of the edge. A sufficient amount of normalized output difference is secured.

According to the second embodiment, in addition to the effect (1) of the first embodiment, the following effect (10) can be obtained.
(10) The focal length of the microlens 41 corresponding to the incident light of the color component on the relatively long wavelength side and the focal length of the microlens 41 corresponding to the incident light of the color component on the relatively short wavelength side are mutually set. By making it different, it is possible to easily realize a configuration that enlarges the difference in the color component amount between the color components of the edges to be compared when the in-focus state is detected.

(Third embodiment)
Next, a third embodiment will be described. The third embodiment is different from the first embodiment in that a motor M is provided as an example of an element driving unit that drives the image sensor 13 in the optical axis direction of incident light. Therefore, in the following description, a configuration that is different from the first embodiment will be mainly described, and a configuration that is the same as or equivalent to that of the first embodiment will be denoted by the same reference numeral, and redundant description will be omitted.

  In the present embodiment, the imaging device 13 does not include the convex pixel 40RA as the red pixel 40R constituting the imaging pixel 40, and the photoelectric conversion surfaces of the photoelectric conversion units 43R of all the red pixels 40R are incident light. They are arranged at the same position in the optical axis direction. Further, the imaging element 13 does not include the concave pixel 40BA as the blue pixel 40B constituting the imaging pixel 40, and the photoelectric conversion surfaces of the photoelectric conversion units 43B of all the blue pixels 40B are the same in the optical axis direction of the incident light. It is arranged at the position. That is, in the imaging element 13, the photoelectric conversion surfaces of the photoelectric conversion units 43 of all the imaging pixels 40 are arranged at the same position in the optical axis direction of incident light.

  Here, in FIG. 21A, when the image picked up by the image pickup device 13 configured as described above is in a front-pin state with respect to the subject, the incident light that has passed through the focus lens 11 shows the subject. A state in which the position of the image sensor 13 at the time of imaging is varied for each color component of incident light is shown. As shown in FIG. 21A, when the image picked up by the image pickup device 13 is in a front pin state with respect to the subject, incident light that has passed through the focus lens 11 is changed to the image of the subject picked up by the image pickup device 13. The amount of focus gradually increases in the order of blue (B) light, green (G) light, and red (R) light.

  In the present embodiment, when the incident light that has passed through the focus lens 11 images the subject, the image sensor 13 is driven in the optical axis direction of the incident light based on the driving force transmitted from the motor M. Then, the focus detection unit 23 of the MPU 16 is arranged in the second position (the position on the left side in FIG. 21A) by driving the imaging element 13 in the direction approaching the focus lens 11 in the optical axis direction of the incident light. In this state, the red pixel signal is selectively read out of the pixel signals of the subject captured by the incident light of each color that has passed through the focus lens 11. Then, the focus detection unit 23 of the MPU 16 uses the read red pixel signal for calculation of the defocus feature amount of the subject.

  In this case, the defocus amount of the subject imaged by the red (R) light on the image sensor 13 is larger than when the image sensor 13 is not driven in the optical axis direction of the incident light. As a result, the difference between the defocus amount of the subject imaged by the red (R) light on the image sensor 13 and the defocus amount of the subject imaged by the green (G) light on the image sensor 13 is determined by the image sensor 13. This is larger than the case where it is not driven in the optical axis direction of the incident light.

  Further, the focus detection unit 23 of the MPU 16 is disposed in the first position (right side in FIG. 21A) by driving the imaging element 13 in the direction away from the focus lens 11 in the optical axis direction of the incident light. Thus, the blue pixel signal is selectively read out of the pixel signals of the subject imaged by the incident light of each color that has passed through the focus lens 11. Then, the focus detection unit 23 of the MPU 16 uses the read blue pixel signal for calculation of the defocus feature amount of the subject.

  In this case, the defocus amount of the subject imaged by the blue (B) light on the image sensor 13 is smaller than when the image sensor 13 is not driven in the optical axis direction of the incident light. As a result, the difference between the defocus amount of the subject imaged by the blue (B) light on the image sensor 13 and the defocus amount of the object imaged by the green (G) light on the image sensor 13 is determined by the image sensor 13. This is larger than the case where it is not driven in the optical axis direction of the incident light.

  FIG. 21B shows the position of the image sensor 13 when incident light that has passed through the focus lens 11 images the subject when the image captured by the image sensor 13 is in a rear-pin state with respect to the subject. A state in which the color components of the incident light are made different is shown. As shown in FIG. 21B, when the image picked up by the image pickup device 13 is in a back-pinned state with respect to the subject, the incident light that has passed through the focus lens 11 is changed to the image of the subject picked up by the image pickup device 13. The amount of focus gradually increases in the order of red (R) light, green (G) light, and blue (B) light.

  In the present embodiment, when the incident light that has passed through the focus lens 11 images the subject, the image sensor 13 is driven in the optical axis direction of the incident light based on the driving force transmitted from the motor M. Then, the focus detection unit 23 of the MPU 16 is driven in the direction in which the imaging element 13 approaches the focus lens 11 in the optical axis direction of the incident light and is disposed at the second position (left side in FIG. 21B). Thus, the red pixel signal is selectively read out from the pixel signals of the subject imaged by the incident light of each color that has passed through the focus lens 11. Then, the focus detection unit 23 of the MPU 16 uses the read red pixel signal for calculation of the defocus feature amount of the subject.

  In this case, the defocus amount of the subject imaged by the red (R) light on the image sensor 13 is smaller than when the image sensor 13 is not driven in the optical axis direction of the incident light. As a result, the difference between the defocus amount of the subject imaged by the red (R) light on the image sensor 13 and the defocus amount of the subject imaged by the green (G) light on the image sensor 13 is determined by the image sensor 13. This is larger than the case where it is not driven in the optical axis direction of the incident light.

  In addition, the focus detection unit 23 of the MPU 16 is driven in a direction away from the focus lens 11 in the optical axis direction of the incident light and is disposed at the first position (right side in FIG. 21B). Thus, the blue pixel signal is selectively read out of the pixel signals of the subject imaged by the incident light of each color that has passed through the focus lens 11. Then, the focus detection unit 23 of the MPU 16 uses the read blue pixel signal for calculation of the defocus feature amount of the subject.

  In this case, the defocus amount of the subject imaged by the blue (B) light on the image sensor 13 is larger than when the image sensor 13 is not driven in the optical axis direction of the incident light. As a result, the difference between the defocus amount of the subject imaged by the blue (B) light on the image sensor 13 and the defocus amount of the object imaged by the green (G) light on the image sensor 13 is determined by the image sensor 13. This is larger than the case where it is not driven in the optical axis direction of the incident light.

  Therefore, even when the defocus amount of the edge included in the through image is small, red (R) and green (G) or blue (B) and green (G) color components in the vicinity of the edge. A sufficient amount of normalized output difference is secured.

According to the third embodiment, in addition to the effect (1) of the first embodiment, the following effect (11) can be obtained.
(11) The position of the image sensor 13 that captures the edge of the subject for detecting the in-focus state of the incident light of the color component on the relatively long wavelength side, and the incident light of the color component on the relatively short wavelength side In order to detect the in-focus state, the position of the image sensor 13 that images the edge of the subject is made different from each other. Therefore, it is possible to easily realize a configuration that enlarges the difference in the color component amount between the color components of the edges to be compared when detecting the in-focus state.

(Fourth embodiment)
Next, a fourth embodiment will be described. Note that the fourth embodiment is different from the first embodiment in that the imaging element 13 converts incident light of a plurality of color components into electric charges using a common imaging pixel. Therefore, in the following description, a configuration that is different from the first embodiment will be mainly described, and a configuration that is the same as or equivalent to that of the first embodiment will be denoted by the same reference numeral, and redundant description will be omitted.

FIGS. 22A to 22C show an example of the image sensor 13 including, as the imaging pixel 40, a white pixel 40W that converts incident light of a plurality of color components into electric charges.
As shown in FIG. 22A, the imaging element 13R of the comparative example includes a color filter 42W through which the white pixel 40W selectively transmits light in the visible light wavelength region. In the white pixel 40W, photoelectric conversion units 43WB, 43WG, and 43WR are stacked in three layers. Of the incident light incident on the image sensor 13, light in the visible wavelength region is collected by the microlens 41W onto the photoelectric conversion units 43WB, 43WG, and 43WR. That is, the light condensed on the photoelectric conversion units 43WB, 43WG, and 43WR includes red, green, and blue color component lights. The incident light of each color is converted into electric charge in the photoelectric conversion units 43WB, 43WG, and 43WR of the corresponding layers. Specifically, the blue color component light is converted into electric charge in the uppermost photoelectric conversion unit 43WB, and the green color component light is converted into electric charge in the intermediate layer photoelectric conversion unit 43WG. The light of the red color component is converted into electric charge in the lowermost photoelectric conversion unit 43WR.

  On the other hand, in the present embodiment, the image pickup device 13A in the example shown in FIG. 22B selectively transmits light in the visible light wavelength region in the same manner as the image pickup device 13 in the comparative example. And the photoelectric conversion units 43WB, 43WG, and 43WR are stacked in three layers. The light of the color component of each color is converted into electric charge in the photoelectric conversion units 43WB, 43WG, and 43WR of the corresponding layers. However, the thickness of the uppermost photoelectric conversion unit 43WB is thinner than that of the imaging element 13R of the comparative example.

  In the present embodiment, the image sensor 13B in the example illustrated in FIG. 22C is a color in which the white pixels 40W selectively transmit light in the visible wavelength region, similarly to the image sensor 13R in the comparative example. A filter 42W is provided, and photoelectric conversion units 43WB, 43WG, and 43WR are stacked in three layers. The light of the color component of each color is converted into electric charge in the photoelectric conversion units 43WB, 43WG, and 43WR of the corresponding layers. However, the thickness of the photoelectric conversion unit 43WG of the intermediate layer is thinner than that of the imaging element 13R of the comparative example.

  FIG. 23A shows photoelectric conversion of incident light of each color component that has passed through the microlens 41W when the image picked up by the image pickup device 13R of the comparative example is in the front pin state with respect to the subject. A state is shown in which light is incident on the units 43WB, 43WG, and 43WR to image a subject. As shown in FIG. 23A, when the image picked up by the image pickup device 13R of the comparative example is in the front pin state with respect to the subject, the light passing through the microlens 41W is converted into a photoelectric conversion unit for each color component. The defocus amount of the subject imaged on 43WB, 43WG, and 43WR gradually increases in the order of blue (B) light, green (G) light, and red (R) light.

  On the other hand, as shown in FIG. 23B, in the present embodiment, the imaging device 13A in the example shown in FIG. 22B has the uppermost photoelectric conversion in which the light of the blue color component is converted into the electric charge. The photoelectric conversion surface of the portion 43WB is located on the exit side (opposite to the microlens 41W) in the optical axis direction of the incident light as compared with the imaging element 13R of the comparative example. Therefore, the defocus amount of the subject in which the light of the blue color component is imaged on the uppermost photoelectric conversion unit 43WB is smaller than that of the imaging element 13R of the comparative example. As a result, the defocus amount of the subject imaged by the blue (B) light on the uppermost photoelectric conversion unit 43WB, and the defocus amount of the subject imaged by the green (G) light on the photoelectric conversion unit 43WG of the intermediate layer Is larger than that of the imaging element 13R of the comparative example. In this case, the normalized output of the blue (B) color component amount in the vicinity of the edge imaged by the uppermost photoelectric conversion unit 43WB and the green (G) color in the vicinity of the edge imaged by the intermediate layer photoelectric conversion unit 43WG. The difference between the color component amount and the normalized output is larger than that of the imaging element 13R of the comparative example. Therefore, even when the defocus amount of the edge included in the through image is small, a sufficient difference between the normalized output of the blue (B) and green (G) color component amounts in the vicinity of the edge is ensured. The

  Further, as shown in FIG. 23C, in the present embodiment, the imaging device 13B in the example shown in FIG. 22C has the lowermost photoelectric conversion unit 43WR that converts red color component light into electric charge. The photoelectric conversion surface is positioned on the incident side (microlens 41W side) in the optical axis direction of the incident light as compared with the imaging element 13R of the comparative example. Therefore, the defocus amount of the subject in which the light of the red color component is imaged on the lowermost photoelectric conversion unit 43WR is larger than that in the imaging element 13R of the comparative example. As a result, the defocus amount of the subject imaged by the red (R) light on the lowermost photoelectric conversion unit 43WR and the defocus amount of the subject imaged by the green (G) light on the photoelectric conversion unit 43WG of the intermediate layer Is larger than that of the imaging element 13R of the comparative example. In this case, the normalized output of the color component amount of red (R) in the vicinity of the edge imaged by the photoelectric conversion unit 43WR in the lowermost layer and the green (G) in the vicinity of the edge imaged by the photoelectric conversion unit 43WG in the intermediate layer. The difference between the color component amount and the normalized output is larger than that of the imaging element 13R of the comparative example. Therefore, even when the defocus amount of the edge included in the through image is small, a sufficient difference between the normalized output of the red (R) and green (G) color component amounts in the vicinity of the edge is ensured. The

  In FIG. 24A, when the image picked up by the image pickup device 13R of the comparative example is in a back-pin state with respect to the subject, incident light of each color component that has passed through the microlens 41W is converted into the photoelectric conversion unit 43WB. , 43WG and 43WR are shown in a state where the subject is imaged. As shown in FIG. 24A, when the image picked up by the image pickup device 13R of the comparative example is in a back-pin state with respect to the subject, the light passing through the microlens 41W is converted into a photoelectric conversion unit for each color component. The defocus amount of the subject imaged on 43WB, 43WG, and 43WR gradually increases in the order of red (R) light, green (G) light, and blue (B) light.

  On the other hand, as shown in FIG. 24B, in the present embodiment, the imaging device 13A in the example shown in FIG. 22B has the uppermost photoelectric conversion in which light of a blue color component is converted into electric charge. The photoelectric conversion surface of the portion 43WB is located on the exit side (opposite to the microlens 41W) in the optical axis direction of the incident light as compared with the imaging element 13R of the comparative example. Therefore, the defocus amount of the subject in which the light of the blue color component is imaged on the uppermost photoelectric conversion unit 43WB is larger than that in the imaging element 13R of the comparative example. As a result, the defocus amount of the subject imaged by the blue (B) light on the uppermost photoelectric conversion unit 43WB, and the defocus amount of the subject imaged by the green (G) light on the photoelectric conversion unit 43WG of the intermediate layer Is larger than that of the imaging element 13R of the comparative example. In this case, the normalized output of the blue (B) color component amount in the vicinity of the edge imaged by the uppermost photoelectric conversion unit 43WB and the green (G) color in the vicinity of the edge imaged by the intermediate layer photoelectric conversion unit 43WG. The difference between the color component amount and the normalized output is larger than that of the imaging element 13R of the comparative example. Therefore, even when the defocus amount of the edge included in the through image is small, a sufficient difference between the normalized output of the blue (B) and green (G) color component amounts in the vicinity of the edge is ensured. The

  In addition, as shown in FIG. 24C, in this embodiment, the imaging device 13B in the example shown in FIG. 22C has the lowermost photoelectric conversion unit 43WR that converts red color component light into electric charge. The photoelectric conversion surface is positioned on the incident side (microlens 41W side) in the optical axis direction of the incident light as compared with the imaging element 13R of the comparative example. For this reason, the defocus amount of the subject in which the light of the red color component is imaged on the photoelectric conversion unit 43WR in the lowermost layer is smaller than that in the case of the imaging element 13R of the comparative example. As a result, the defocus amount of the subject imaged by the red (R) light on the lowermost photoelectric conversion unit 43WR and the defocus amount of the subject imaged by the green (G) light on the photoelectric conversion unit 43WG of the intermediate layer Is larger than that of the imaging element 13R of the comparative example. In this case, the normalized output of the color component amount of red (R) in the vicinity of the edge imaged by the photoelectric conversion unit 43WR in the lowermost layer and the green (G) in the vicinity of the edge imaged by the photoelectric conversion unit 43WG in the intermediate layer. The difference between the color component amount and the normalized output is larger than that of the image sensor 13 of the comparative example. Therefore, even when the defocus amount of the edge included in the through image is small, a sufficient difference between the normalized output of the red (R) and green (G) color component amounts in the vicinity of the edge is ensured. The

  Therefore, even when the defocus amount of the edge included in the through image is small, red (R) and green (G) or blue (B) and green (G) color components in the vicinity of the edge. A sufficient amount of normalized output difference is secured.

According to the fourth embodiment, in addition to the effect (1) of the first embodiment, the following effect (12) can be obtained.
(12) The photoelectric conversion surface on which the incident light of the color component on the relatively long wavelength side is converted into electric charge is located on the incident side in the optical axis direction of the incident light as compared with the conventional case. In addition, the photoelectric conversion surface on which incident light of a color component on the relatively short wavelength side is converted into electric charge is located on the exit side in the optical axis direction of incident light as compared with the conventional case. Therefore, it is possible to easily realize a configuration that enlarges the difference in the color component amount between the color components of the edges to be compared when detecting the in-focus state.

In addition, you may change said each embodiment into another embodiment as follows.
In the first embodiment, as shown in FIG. 25A, the convex pixel 40RA and the concave pixel 40BA are arranged in five areas R11 to R15 in the center, the top, bottom, left, and right among the imaging pixels 40 of the imaging element 13. It may be included in the pixel group located.

In the first embodiment, the convex pixels 40RA and the concave pixels 40BA may be irregularly arranged on the imaging surface of the imaging element 13, as shown in FIG.
In this configuration, even if a captured image is generated using pixel signals captured by the convex pixel 40RA and the concave pixel 40BA, it is possible to suppress the occurrence of moire in the generated captured image.

  In the second embodiment, the red pixel 40R including the micro lens 41RA and the blue pixel 40B including the micro lens 41BA are included in the pixel group located in the areas R11 to R15 illustrated in FIG. Also good. In addition, the red pixel 40R including the microlens 41RA and the blue pixel 40B including the microlens 41BA may be irregularly arranged on the imaging surface of the imaging element 13.

  In the fourth embodiment, the image sensor 13 includes at least the white pixel 40W illustrated in FIG. 22A, the white pixel 40W illustrated in FIG. 22B, and the white pixel 40W illustrated in FIG. A configuration including some combinations as the imaging pixels 40 may be used. In this case, the white pixels 40W shown in FIGS. 22B and 22C may be included in the pixel group located in the areas R1 to R6 shown in FIG. 4 as an example, or FIG. May be included in the pixel group located in the areas R11 to R15. Further, the white pixels 40W illustrated in FIGS. 22B and 22C may be irregularly arranged on the imaging surface of the imaging element 13.

  In the first embodiment, the imaging element 13 does not necessarily have a uniform amount of depression of the concave pixel 40BA. The concave pixel 40BA having a relatively small amount of depression and a concave pixel having a relatively large amount of depression. 40BA may be included. Similarly, the imaging element 13 does not necessarily have a uniform protruding amount of the convex pixel 40RA, and includes a protruding pixel 40RA having a relatively small protruding amount and a protruding pixel 40RA having a relatively large protruding amount. It may be configured.

  In the first embodiment, the image generation unit 24 of the MPU 16 uses the pixel signal of the subject imaged by the concave pixel 40BA when the luminance of the subject imaged by the image sensor 13 is equal to or higher than a predetermined threshold. In this case, when the luminance of the subject is high, incident light with high light intensity is incident and flare or ghost is generated, so that the imaging element 13 is viewed from an angle where the incident light is not intended. This is because incident light is less likely to enter the concave pixel 40BA than the convex pixel 40RA and the flat pixel.

  In the first embodiment, the image generation unit 24 of the MPU 16 presets whether or not the pixel signal of the subject imaged by the concave pixel 40BA is used for the captured image as an initial setting without a determination process. Also good.

  In the first embodiment, the image generation unit 24 of the MPU 16 reads the pixel signal read from the convex pixel 40RA or the concave pixel 40BA without performing the magnification correction process on the pixel signal. A captured image may be generated using a pixel signal.

  In the first embodiment, the image generation unit 24 of the MPU 16 performs shading using a common shading coefficient for the pixel signal read from the flat pixel and the pixel signal read from the convex pixel 40RA or the concave pixel 40BA. Correction processing may be performed.

  In the first embodiment, the image generation unit 24 of the MPU 16 does not read out the pixel signal from the convex pixel 40RA or the concave pixel 40BA, and the through image is generated from the frame image generated based on the pixel signal read from the flat pixel. May be configured. Then, the display control unit 25 of the MPU 16 may cause the monitor 21 to display a through image that does not include a frame image generated using the pixel signal read from the convex pixel 40RA or the concave pixel 40BA.

  In this case, the image generation unit 24 of the MPU 16 first generates an evaluation image based on the pixel signal read from the imaging pixel 40 including the convex pixel 40RA or the concave pixel 40BA, separately from the through image displayed on the monitor 21. To do. Subsequently, the focus detection unit 23 of the MPU 16 calculates an evaluation value of axial chromatic aberration at the edge detected from the evaluation image. As a result, the focus detection unit 23 of the MPU 16 can calculate the edge defocus feature value in the evaluation image as the edge defocus feature value in the through image based on the calculated evaluation value.

In the first embodiment, the image sensor 13 may have a configuration having only one of the convex pixel 40RA and the concave pixel 40BA as the imaging pixel 40.
In the second embodiment, the imaging element 13 includes the convex pixel 40RA as the red pixel 40R that constitutes the imaging pixel 40, and the concave pixel 40BA as the blue pixel 40B that constitutes the imaging pixel 40. May be. The micro lens 41R of the convex pixel 40RA may have a lens surface with a smaller curvature than the micro lens 41R of the other red pixel 40R. Further, the micro lens 41B of the concave pixel 40BA may have a lens surface with a larger curvature than the micro lens 41B of the other blue pixel 40B.

  In the third embodiment, the imaging element 13 includes the convex pixel 40RA as the red pixel 40R that constitutes the imaging pixel 40, and the concave pixel 40BA as the blue pixel 40B that constitutes the imaging pixel 40. May be.

  In the third embodiment, the image sensor 13 captures incident light of all color components (red, green, and blue) that have passed through the focus lens 11 at a plurality of positions that are different from each other in the optical axis direction of the incident light. The pixel signal of the subject to be read may be read out.

  In this case, when the image sensor 13 is disposed at the second position on the focus lens 11 side in the optical axis direction of the incident light, the red pixel signal of the subject pixel signals read from the image sensor 13 is the subject. This is used to calculate the defocus feature amount of. Further, when the image sensor 13 is driven to the side opposite to the focus lens 11 in the optical axis direction of the incident light starting from the second position, a green pixel among the pixel signals of the subject read from the image sensor 13 The signal is used to calculate the defocus feature amount of the subject. Further, when the image sensor 13 is further driven to the side opposite to the focus lens 11 in the optical axis direction of the incident light, among the pixel signals of the subject read out from the image sensor 13, the blue pixel signal is the subject image. Used to calculate the focus feature value.

  In the first to third embodiments, the image sensor 13 selectively transmits light of green (G), yellow (Y), magenta (M), and cyan (C) color components, respectively. The structure provided with the complementary color filter may be sufficient. The imaging device 13 may have a configuration including a filter that selectively transmits light in the wavelength region of the infrared region.

  In the fourth embodiment, the image sensors 13A and 13B include black pixels including the microlens 41W and the photoelectric conversion units 43WB, 43WG, and 43WR stacked in three layers without including the color filter 42W. Instead of the white pixel 40W, a configuration may be employed.

  In each of the embodiments described above, the image sensors 13, 13A, 13B of the embodiments may be newly provided as AF sensors separately from the image sensors 13, 13A, 13B provided for image capturing. .

In each of the above embodiments, the focus detection unit 23 of the MPU 16 may calculate the defocus feature amount of the edge detected from the moving image when shooting the moving image.
In each of the above embodiments, the imaging device is not limited to a digital camera, and may be another imaging device equipped with an image shooting function, such as a video camera, a personal computer, a mobile phone, or a portable game machine. May be.

  DESCRIPTION OF SYMBOLS 10 ... Digital camera as an example of an imaging device, 11 ... Focus lens as an example of a lens, 13, 13A, 13B ... Imaging element, 21 ... Monitor as an example of a display part, 22 ... Edge detection part, 23 ... Focusing Detection unit, 24 ... image generation unit, 25 ... display control unit, 40 ... imaging pixel, 40RA ... convex pixel as an example of first imaging pixel, 40BA ... concave pixel as an example of second imaging pixel, 41 ... condensing Microlens as an example of a lens, 43... Photoelectric conversion unit, F, F1, F2... Frame image, M.

Claims (13)

An image sensor for imaging a subject by incident light incident through a lens;
An edge detection unit that detects, for each color component, an edge of the subject imaged by the imaging element;
A focus detection unit that detects a focus state of the edge of the subject based on a difference or ratio of color component amounts between color components at the edge detected by the edge detection unit, and
An imaging mode of the subject with respect to the imaging device by the incident light is set for each color component of the incident light so that a difference in color component amount between the color components at the edge detected by the edge detection unit is enlarged. An imaging apparatus characterized by comprising:   The photoelectric conversion unit of the imaging pixel of the image sensor to which the incident light of the color component on the relatively long wavelength side is incident is connected to the photoelectric conversion unit of the image sensor on which the incident light of the color component on the relatively short wavelength side is incident. The imaging apparatus according to claim 1, wherein the imaging apparatus is disposed closer to the lens in the optical axis direction of the incident light than the photoelectric conversion unit of the imaging pixel.   The image pickup pixel of the image pickup device has the photoelectric conversion unit in the optical axis direction of the incident light as compared with other image pickup pixels among the image pickup pixels on which the incident light of the color component on the relatively long wavelength side is incident. Compared to the other imaging pixels among the imaging pixels to which the first imaging pixels arranged to protrude toward the lens and the incident light of the color component on the relatively short wavelength side are incident, the photoelectric conversion unit The imaging apparatus according to claim 2, wherein the imaging apparatus includes at least one of second imaging pixels that are recessed from the lens in the direction of the optical axis of the incident light.   The imaging apparatus according to claim 3, further comprising an image generation unit that generates a captured image using image data captured by at least one of the first imaging pixel and the second imaging pixel.   The image generation unit performs image processing different from image data captured by other imaging pixels in the imaging element on image data captured by at least one of the first imaging pixel and the second imaging pixel. The imaging apparatus according to claim 4, wherein the captured image is generated.   The image generation unit performs, as the image processing, shading correction processing using a shading coefficient that is different from the image data captured by the other imaging pixels in the imaging element for the image data captured by the second imaging pixel. The imaging apparatus according to claim 5, wherein the captured image is generated.   The image generation unit generates the captured image after performing a magnification correction process on the image data captured by at least one of the first imaging pixel and the second imaging pixel as the image processing. The imaging device according to claim 5 or 6.   The at least one of the first image pickup pixel and the second image pickup pixel is irregularly arranged on the image pickup pixels of the image pickup element arranged two-dimensionally. The imaging device according to any one of the above.   The display control unit according to claim 1, further comprising: a display control unit configured to display an image captured by an imaging pixel including at least one of the first imaging pixel and the second imaging pixel in the imaging element as a display image on a display unit. The imaging device according to any one of claims 3 to 8. The display image is a through image or a moving image,
The display control unit displays the display image including an image captured by an imaging pixel including at least one of the first imaging pixel and the second imaging pixel in the imaging element as a time-series continuous frame image. The imaging device according to claim 9, wherein the imaging device is displayed on a screen. The imaging pixel of the imaging element has a condensing lens that condenses the incident light on a photoelectric conversion unit,
The imaging pixel in which the incident light of the color component on the relatively long wavelength side is incident on the focal length of the condenser lens of the imaging pixel on which the incident light of the color component on the relatively short wavelength side is incident The imaging device according to claim 1, wherein the imaging device is smaller than a focal length of the condenser lens. An element drive unit that drives the image sensor in the optical axis direction of the incident light;
When the subject for detecting the in-focus state is picked up by the image pickup device by the incident light of the color component having a relatively short wavelength side, the element driving unit moves the image pickup device of the incident light. When the subject for detecting the in-focus state of the edge is picked up by the imaging device by the incident light of the color component on the relatively long wavelength side while being arranged at the first position in the optical axis direction The image pickup device is arranged at a second position located on the lens side in the optical axis direction of the incident light from the first position. The imaging device described in 1. It includes an imaging pixel in which incident light of each color component is converted into electric charge,
In the imaging pixel, the photoelectric conversion unit of the imaging pixel in which the incident light of the color component on the relatively long wavelength side enters the photoelectric of the imaging pixel in which the incident light of the color component on the relatively short wavelength side enters. An imaging device, wherein the imaging device is arranged on an incident side in an optical axis direction of the incident light with respect to the conversion unit.

Images