JP2017102228A - Imaging apparatus and method for controlling the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus capable of performing highly accurate focus detection, according to the lens pupil distance of a wider range, in comparison with a conventional technology and to provide a method for controlling the imaging apparatus.SOLUTION: The imaging apparatus includes a beam splitter 103 which divides a luminous flux after passing through the exit pupil of a photographic lens 500 into a plurality of luminous fluxes, an imaging element 101 which has a plurality of pixel parts having a microlens and a plurality of photoelectric conversion parts and receives one of the divided luminous fluxes, and an imaging element 102. The length of a sensor pupil distance which is a distance from an image surface to a sensor pupil surface located in a point where the principal ray of the pixel part crosses the optical axis of the luminous flux passing through the photographic lens, in the imaging element 101 is shorter than that in the imaging element 102. A camera CPU 104 calculates a defocus amount by using two signals having a parallax obtained from at least one of the imaging element 101 and the imaging element 102.SELECTED DRAWING: Figure 4

Description

  The present invention relates to an imaging apparatus capable of focus detection and a control method thereof.

  In recent years, there has been proposed an imaging apparatus including a plurality of imaging elements that acquire a recording or display image and perform a phase difference type focus detection on an imaging surface, that is, a function suitable for so-called imaging surface phase difference detection. ing. The image pickup apparatus disclosed in Patent Document 1 discloses that the image pickup device has two image pickup devices, that is, an image pickup device for still image shooting and an image pickup device for moving image shooting. Here, when a still image is generated as a recording image using a signal from an image pickup device for still image shooting, a signal acquired from the image pickup device for moving image shooting is used for focus detection. In addition, it is disclosed that when a moving image is generated as a recording image using a signal from an imaging element for moving image shooting, a signal acquired from the imaging element for still image shooting is used for focus detection.

Japanese Patent Laid-Open No. 2015-34917

  In the imaging surface phase difference detection imaging device, the lens pupil distance (exit pupil distance) of the photographing lens and the distance of the surface where the pupil of the received light beam of each pixel of the imaging device substantially coincides (also referred to as sensor pupil distance). The closer it is to (described later in detail), the higher the focus detection accuracy. On the other hand, since the lens pupil distance changes depending on the type of lens and the zoom position, when the deviation between the lens pupil distance and the sensor pupil distance is large, the focus detection accuracy is lowered as compared with the case where the deviation is smaller. For this reason, the range of the lens pupil distance that can perform focus detection with high accuracy with respect to the sensor pupil distance is limited.

  In the above-described Patent Document 1, it is described that an image sensor for still image shooting and an image sensor for moving image are properly used depending on whether a still image or a moving image is acquired as a recording image. . However, there remains a problem with the limited range of lens pupil distances that can be used for focus detection with high accuracy with respect to the sensor pupil distance. A technique capable of performing focus detection has been demanded.

  Accordingly, an object of the present invention is to provide an imaging apparatus capable of performing focus detection with high accuracy corresponding to a wider range of lens pupil distances and a control method therefor as compared with the prior art.

  The present invention relates to a light beam splitting unit that splits a light beam that has passed through a photographing lens into a plurality of light beams, and a pixel unit that includes a plurality of photoelectric conversion units that receive and photoelectrically convert light beams having parallax for one microlens. And a first imaging element that receives one of the divided light beams, and a pixel unit that includes a plurality of photoelectric conversion units that respectively receive and photoelectrically convert light beams having parallax with respect to one microlens. A second imaging element that receives the other of the divided luminous fluxes and has a sensor pupil distance longer than the sensor pupil distance of the first imaging element; and the first imaging element; And a first focus detection unit configured to calculate a defocus amount using two signals having parallax acquired from at least one of the second imaging elements.

  Further, the present invention provides a pixel having a light beam splitting unit that splits a light beam that has passed through the photographing lens into a plurality of light beams, and a plurality of photoelectric conversion units that receive and photoelectrically convert each of the light beams having parallax for one microlens. A plurality of pixel units, a second imaging element that receives one of the divided light beams, and a pixel unit that includes a photoelectric conversion unit that receives and photoelectrically converts the light beam, and the other of the divided light beams A third image sensor that receives light; and first focus detection means that calculates a defocus amount using a signal acquired from at least one of the second image sensor and the third image sensor; When the reliability of the signal acquired from the second image sensor is high, the first focus detection unit uses the signal acquired from the second image sensor more than the signal acquired from the third image sensor. Use focus detection often On the other hand, when the reliability of the signal acquired from the second image sensor is low, the first focus detection means uses the signal acquired from the third image sensor as compared with the signal acquired from the second image sensor. Is also used to perform focus detection.

  According to the present invention, it is possible to perform focus detection with high accuracy corresponding to a wider range of lens pupil distances than in the prior art.

Configuration diagram of imaging device Configuration diagram of the image sensor 101 Readout circuit configuration diagram of the image sensor 101 The figure explaining the exit pupil distance (lens pupil distance) PL and sensor pupil distance of the taking lens 500 The figure explaining the relative positional relationship of the focus detection pupil of each pixel part, and the exit pupil of the photographic lens 500 Diagram explaining shading characteristics The figure explaining the change with respect to the lens pupil distance of the shading ratio SH The figure explaining the example of a waveform of a focus detection signal The figure explaining the relationship between the phase difference of a pair of 2 images, and a correlation value A diagram for explaining shading correction information The figure explaining defocus amount conversion information Main flow chart for shooting Subroutine flowchart of “focus detection 1” and “still image shooting 1” of the image sensor 101 Subroutine flowchart of “focus detection 2” and “still image shooting 2” of the image sensor 102 Subroutine flowchart of “focus detection 3” and “focus detection 4” in the second embodiment Configuration diagram of image pickup device in embodiment 3 FIG. 6 is a diagram for explaining sensor pupil distances according to the third embodiment. Configuration diagram of image pickup device in embodiment 4 FIG. 6 is a diagram for explaining sensor pupil distances according to the fourth embodiment. Subroutine flowchart of “focus detection 5” and “focus detection 6” in the fourth embodiment The figure explaining the modification in Examples 1-4

  Hereinafter, preferred embodiments to which the present invention is applied will be described with reference to the accompanying drawings.

[Example 1]
[Configuration of camera body]
FIG. 1 is a configuration diagram of an interchangeable lens digital camera main body 100 (hereinafter, camera main body 100) and a photographing lens 500 as an example of an imaging apparatus embodying the present invention.

  The photographing lens 500 can be attached to and detached from the camera body 100, and the light beam transmitted through each lens group in the photographing lens 500 is incident on a beam splitter 103 as a light beam dividing unit provided in the camera body 100. The beam splitter 103 is fixed in the camera body 100, and is a half mirror in this embodiment. One of the light beams divided by the beam splitter 103 passes through the beam splitter 103 and is guided to the image sensor 101 (first image sensor) arranged to form a subject image on the first image pickup surface. . The other light beam is reflected by the beam splitter 103 and guided to the image sensor 102 (second image sensor) arranged so as to form a subject image on the second imaging surface. Here, the imaging surface is a sensor surface of the imaging device. Details will be described later. The beam splitter 103 may not be a half mirror as long as it can split an incident light beam in the same manner as a half mirror.

  The first and second imaging surfaces are at optically equivalent (conjugated) positions when viewed from the photographic lens. In other words, the image pickup device 101 arranged on the first image pickup surface and the image pickup device 102 arranged on the second image pickup surface are respectively optically conjugate with respect to the subject via the photographing lens 500. It is in.

  A subject image having a brightness corresponding to the transmittance and reflectance of the beam splitter 103 is formed on the first and second imaging surfaces. Although it is desirable that the half mirror disposed in the imaging light beam is an ideal plane and the refractive index of the region through which the light beam is transmitted is uniform, it may not actually be the case. For this reason, an image formed by a light beam transmitted or reflected by the beam splitter 103 may be deteriorated in image quality as compared with a case where the image is not transmitted or reflected by the beam splitter 103. When the half mirror is made of thin glass, the degree of image quality degradation is relatively large in the image formed by the reflected light beam compared to the image formed by the transmitted light beam. Therefore, in this embodiment, the transmissive-side image sensor 101 is used as an image sensor for capturing a still image, and the reflective-side image sensor 102 is used as an image sensor used for capturing a moving image. However, the present invention is not limited to this mode, and the positions of the image sensor 101 and the image sensor 102 may be interchanged according to the characteristics of the beam splitter 103 and other conditions.

  The image sensor 101 and the image sensor 102 made up of CMOS area sensors have a plurality of pixel portions arranged in a matrix for converting a subject image into an electric signal. The pixel information converted into the electrical signal is processed by AFE 114 (first image processing unit) and AFE 115 (second image processing unit), which are circuits dedicated to image processing. Specifically, various correction processes for obtaining a recorded image signal and focus detection signal, a process for converting the obtained recorded image signal into a live view image signal and a recorded image signal, and the like are performed. In the present embodiment, since the camera body 100 has AFEs corresponding to the two image sensors, signals acquired from the two image sensors can be processed in parallel. Signals processed by the AFE 114 and the AFE 115 are transmitted to the camera CPU 104. The operation member 105 is various members for setting a camera shooting mode, shooting conditions, and the like. A storage medium 106 is a flash memory, and is a medium for recording captured still images and moving images. The in-finder display 107 includes a display 108 and an eyepiece 109 as a small and high-definition display means such as an organic EL display or a liquid crystal display. As the external display 110, an organic EL display or a liquid crystal display having a screen size suitable for naked-eye viewing is used. Various information such as the setting state of the camera body 100, a live view image, and a captured image are displayed on the in-finder display 107 and the external display 110.

  The focal plane shutter 111 is disposed in front of the image sensor 101. The shutter driving unit 112 is, for example, a motor, and controls the exposure time when capturing a still image by driving and controlling the shutter blades. A camera-side communication terminal 113 is provided on a camera mount (not shown) for mounting the taking lens. The camera side communication terminal 113 transmits and receives information exchanged between the camera CPU 104 and a later-described lens CPU 507 together with the lens side communication terminal 508 provided in the lens mount unit.

  The photographic lens 500 is detachable from the camera body 100, and in this embodiment is a zoom lens having a variable focal length. The luminous flux from the subject passes through the first lens group 501, the second lens group 502, and the third lens group 503, and forms a subject image on the imaging surface in the camera body 100. The second lens group 502 functions as a variator that performs zooming by moving back and forth in the optical axis direction. The third lens group 503 functions as a focus lens that moves forward and backward in the optical axis direction to adjust the focus. The third lens group 503 is driven by a focus driving unit 504 using a stepping motor or the like. The iris diaphragm 505 is composed of a plurality of diaphragm blades for adjusting the amount of light incident on the photographing lens. The diaphragm drive unit 506 drives the iris diaphragm 505 to narrow down the diaphragm blades until a predetermined F number is reached. The lens CPU 507 communicates with the camera CPU 104 via the lens-side communication terminal 508 and the camera-side communication terminal 113 to transmit and receive various types of information, and drives the focus driving unit 504 and the aperture driving unit 506 based on commands from the camera CPU 104. Control.

  Since the photographer selects and uses a photographing lens suitable for the purpose of photographing, the zoom range and the open F number of the photographing lens 500 can take various values. However, in this embodiment, unless otherwise described. The open F number will be described assuming a constant value of F2 regardless of the zoom state or the focus state.

[Configuration of image sensor]
FIG. 2 is a diagram illustrating the configuration of the image sensor 101. The image sensor 101 of the present embodiment is an image sensor that employs a so-called image plane phase difference AF method that performs phase difference type focus detection on the image plane. In the present embodiment, as an example, the image sensor 101 and the image sensor 102 will be described as having the same structure and the same number of pixels except for the amount of eccentricity of a microlens described later. Therefore, description of the image sensor 102 is omitted. However, the image sensor 101 and the image sensor 102 may have different structures and configurations, and examples thereof will be described later in Embodiments 2 to 4.

  FIG. 4A is a plan view of a part of the pixel portion in the vicinity of the center of the imaging surface (near the image height of 0) as viewed from the photographing lens side. The plurality of pixel portions included in the imaging element 101 are square pixel portions each having a size of 4 μm in the horizontal direction (x) and the vertical direction (y) on the imaging surface. The image sensor is an image sensor having 24 million effective pixels, in which these pixel portions are arranged in a horizontal direction of 6000 pixels and a vertical direction of 4000 pixels. The size of the imaging region can be obtained by multiplying the size of the pixel portion, that is, the pixel pitch by the number of pixels. In this case, the size is 24 mm in the horizontal direction and 16 mm in the vertical direction. In each pixel portion, RGB color filters are arranged in a mosaic pattern.

  FIG. 2B is a cross-sectional view of one pixel portion in the pixel group. A silicon substrate 101d that forms a base of the CMOS image sensor includes a photoelectric conversion unit 101a (first photoelectric conversion unit) and a photoelectric conversion unit 101b (second photoelectric conversion unit). In addition, the silicon substrate 101d includes a switching transistor (not shown) that converts the electric charges generated in the photoelectric conversion unit 101a and the photoelectric conversion unit 101b into a voltage and reads them out, and a signal after photoelectric conversion is transmitted by the wiring layer 101e. Read out.

  Each wiring layer 101e is insulated by a transparent interlayer film 101f. A color filter 101g for color separation is provided under the on-chip microlens 101c. The shape of the on-chip microlens 101c is determined so that the focal position thereof substantially coincides with the upper surfaces of the photoelectric conversion unit 101a and the photoelectric conversion unit 101b. Therefore, the photoelectric conversion units 101a and 101b are back-projected through the on-chip microlens 101c to the vicinity of an exit pupil EP of the photographing lens 500 described later, and the backprojected image is used as a focus detection pupil for phase difference focus detection. Function. When performing phase difference type focus detection, the output signals of the photoelectric conversion unit 101a and the photoelectric conversion unit 101b are individually processed to generate a pair of phase difference images. The camera CPU 104 (first focus detection unit) calculates the defocus amount of the subject image on the imaging surface from the relative image shift amount of the two images. In this embodiment, focus detection is to calculate the defocus amount based on the phase difference between two images. Further, the AFE 114 (addition control means) adds the signals of the pair of photoelectric conversion units 101a and 101b to obtain a recorded image signal of a still image or a moving image or an image signal for live view (for display). Note that the addition processing may be performed by providing a dedicated circuit, or after the image pickup device individually outputs the output signals of the photoelectric conversion unit 101a and the photoelectric conversion unit 101b, the addition processing is performed in the image pickup device and output. May be.

  FIG. 3 shows the configuration of the readout circuit of the image sensor 101 of this embodiment. The photoelectric conversion unit 101a and the photoelectric conversion unit 101b are a pair of photoelectric conversion units arranged in one pixel unit, 121 is a horizontal scanning circuit, and 123 is a vertical scanning circuit. In addition, horizontal scanning lines 122a and 122b and vertical scanning lines 124a and 124b are wired at the boundary between the pixel portions, and signals from the photoelectric conversion units are read out to the outside through these scanning lines. Since the image sensor 102 also has a readout circuit similar to this, description thereof is omitted.

  The image sensor 101 and the image sensor 102 of the present embodiment have the following two types of readout modes. The first readout mode is referred to as an all-pixel readout mode, and is a mode for capturing a still image or moving image for recording. In this case, signals of all the pixel portions are read out. The second readout mode is called a thinning readout mode, and is a mode for displaying a live view image having a smaller number of pixels than the recording still image. The live view image is an image for displaying an image acquired by the image sensor on the in-finder display 107 or the external display 110 in real time. Since the number of pixels required for live view is smaller than the total number of pixels, the image sensor reads out signals only from the pixel portion thinned at a predetermined ratio in both the x direction and the y direction, thereby reducing the processing load on the signal processing circuit. It also contributes to reduction of power consumption. In both the first and second readout modes, the signals of the one to two photoelectric conversion units included in each pixel unit can be independently read out. Therefore, the signal for focus detection in any mode. (A signal for defocus amount calculation) can be generated.

  In this embodiment, the image sensor 101 is mainly used for still image shooting, but does not prohibit moving image shooting. For example, during moving image shooting by the image sensor 102, the image sensor 101 can record the thinned-out image described above as a moving image. Similarly, the image sensor 102 is mainly used for moving image shooting, but still image shooting is also possible. For example, it is possible to record one desired frame as a still image during moving image recording.

[Exit pupil / exit pupil distance (lens pupil distance)]
FIG. 4 is a diagram illustrating the exit pupil distance (lens pupil distance) PL1 and the sensor pupil distance PS of the photographing lens 500. 4A is an explanatory diagram of the image sensor 101, and FIG. 4B is an explanatory diagram of the image sensor 102. First, the exit pupil distance (lens pupil distance) PL1 of the photographing lens will be described.

  FIG. 4A illustrates an optical element portion of the photographing lens 500 and two pixel portions 1011 and 1012 included in the image pickup device 101. The pixel unit 1011 is a pixel unit disposed at the center of the imaging surface, that is, the image height x = 0, and the pixel unit 1012 is a pixel unit disposed at a location near the end of the imaging surface, for example, the image height x = 10 mm.

  The photographing lens 500 constituting this embodiment is a zoom lens having a variable focal length, and the focal length, the open F number, and the exit pupil distance PL1 (described later) change according to the zoom operation. In FIG. 4, it is assumed that the focal length of the taking lens 500 is set at the wide angle end. The first lens group 501 constituting the photographing lens 500 has its frontmost surface held by a lens barrel (not shown) with a front frame 501r, and similarly, the rearmost surface of the third lens group 503 is held by a lens barrel with a rear frame 503r. Has been. There is an iris diaphragm 505 between the first lens group 501 and the third lens group 503. The aperture of the iris diaphragm in this figure shows the open state. When the aperture is open, the outermost rays L11a and L11b out of the light beams reaching the pixel unit 1011 become rays regulated by the opening of the iris diaphragm 505, and the angle θ11 formed by these rays is the open F number at the wide angle end. Is the angle of the luminous flux corresponding to.

  On the other hand, the light beam that passes through the photographing lens 500 and reaches the pixel portion 1012 disposed in the vicinity of the end of the imaging surface is a light beam in an area sandwiched between the light beams L12a and L12b. Here, the light beam L12a is a light beam regulated by the front frame 501r, and is a so-called underline. The light beam L12b is a light beam regulated by the rear frame 503r and is a light beam called an overline. The angle θ12 formed by the upper line L12b and the lower line L12a is smaller than the light beam angle θ11 at the center of the screen due to vignetting.

  An intermediate ray L12c between the upper line L12b and the lower line L12a is the principal ray, and the intersection angle of the optical axis L11c and the principal ray L12c forms an angle β1. A plane perpendicular to the optical axis L11c passing through the point CL1 where the optical axis L11c and the principal ray L12c intersect is an exit pupil plane EPL1 on which the exit pupil is located. On the exit pupil plane EPL1, the region through which the light beam reaching the pixel portion 1012 passes is the exit pupil EP (not shown in FIG. 4) viewed from the pixel portion 1012. An interval (distance) between the imaging surface and the exit pupil EP (or exit pupil plane EPL1) is an exit pupil distance PL1. Here, the exit pupil distance PL1 of the photographing lens is also referred to as a lens pupil distance PL1. Since the position of the exit pupil EP changes depending on the zoom and focus state changes of the photographing lens 500, the lens pupil distance PL1 also changes depending on the zoom and focus state changes of the photographing lens 500.

  Note that since the vignetting changes according to the iris diaphragm being narrowed down, the lens pupil distance PL1 of the photographing lens 500 also changes strictly depending on the narrowed state. However, in general, since the lens design is performed so that the variation of the exit pupil distance PL1 due to the narrowing is small, the change in the exit pupil distance due to the narrowing is often negligible. Therefore, in this embodiment as well, the exit pupil distance changes according to the zoom state and the focus state, but will be described as being unchanged with respect to the aperture value.

[Focus detection pupil]
Next, the structure of the pixel portion and the focus detection pupil will be described. The pixel unit 1011 is a pixel unit located at the center of the imaging region, that is, on the optical axis of the photographing lens. The pair of photoelectric conversion units 1011a and 1011b are arranged adjacent to each other, and the boundary portion is a dead zone having a minimum width. The center of the boundary portion coincides with the center of the pixel portion, and the pair of photoelectric conversion portions are symmetrical with respect to the center of the pixel portion in the x direction. Therefore, the gravity center positions of the individual photoelectric conversion units are equidistant in the x direction with respect to the center of the pixel unit. The principal point 1011p is a reference position for considering the light refraction action of the microlens 1011c included in the pixel portion, and a plane that passes through the principal point 1011p and is perpendicular to the optical axis is a so-called imaging surface of the image sensor 101, so-called imaging. It becomes surface IP1. That is, the state where the focus position of the subject image formed by the photographic lens 500 is coincident with the planned imaging plane IP1 connecting the principal points 1011p of each microlens is in focus, and the subject image acquired by the image sensor at this time is in focus. It will be the most advanced. Since the thickness of the microlens is about 1 μm and the distance between the apex of the microlens and the main point 1011p is a smaller value, the main point 1011p can be regarded as the apex of the microlens.

  In the pixel portion 1011, the optical axis of the microlens 1011 c and the center of the pixel portion (boundary center of a pair of photoelectric conversion portions) are coincident. On the other hand, in the pixel portion 1012 located at the end of the imaging surface IP1, the optical axis of the micro lens 1012c and the center of the pixel portion do not coincide with each other, and are decentered by a predetermined amount dx1 near the optical axis of the photographing lens 500. . Here, a line connecting the center of the boundary between the pair of photoelectric conversion units and the principal point of the microlens is defined as a principal ray of the pixel unit. Then, the principal ray of the pixel portion is inclined by a predetermined angle ω1 with respect to the optical axis, and intersects the optical axis at a point CS1 separated from the imaging surface IP1 by a predetermined distance. A virtual plane passing through the intersection CS1 and orthogonal to the optical axis is referred to as a sensor pupil plane SPL1, and a distance PS1 between the sensor pupil plane SPL1 and the imaging plane IP1 is referred to as a sensor pupil distance PS1. On the sensor pupil plane, the focus detection pupils of all the pixel portions substantially coincide. The reason will be described later.

  In the pixel portion 1011 at the center of the imaging surface, the height h1 between the principal point 1011p of the microlens 1011c and the upper surfaces of the pair of photoelectric conversion portions is the optical height of the pixel portion. The strict optical height is a value obtained by multiplying the mechanical dimension height h1 by the refractive index of the optical path portion in the pixel unit. However, for the sake of simplicity of explanation, the illustrated height h1 is used as the pixel unit. Of height. The shape (optical power) of the microlens 1011c is set so that the focal point of the microlens 1011c substantially coincides with the upper surfaces of the photoelectric conversion units 1011a and 1011b. Here, in FIG. 4, the lens pupil distance PL1 and the sensor pupil distance PS1 are drawn several times as large as the height h1 of the pixel portion. However, in reality, the height h1 of the pixel portion is on the order of micrometers, whereas the lens pupil distance PL1 and the sensor pupil distance PS1 are on the order of several tens of millimeters, and there is a gap of about 4 digits in the size of both. is there. That is, when considering the light condensing action of the microlens, it can be considered that the exit pupil and the sensor pupil plane of the photographing lens viewed from the pixel unit 1011 are very far away. Therefore, if the focal position of the microlens 1011c is substantially coincident with the upper surface of the photoelectric conversion unit, the upper surface of the photoelectric conversion unit is projected on a distant plane, but the size of the projected image is projected. It is proportional to the distance of the surface to be done. Here, if the projection plane is the aforementioned sensor pupil plane SPL1, a pair of back-projected images AP1a and AP1b corresponding to the pair of photoelectric conversion units 1011a and 1011b are formed on the sensor pupil plane SPL1, and this is the focus detection time. It becomes a focus detection pupil that defines the luminous flux. That is, the photoelectric conversion unit 1011a receives a light beam in a region corresponding to the backprojection image AP1a and photoelectrically converts it. In addition, the photoelectric conversion unit 1011b receives and photoelectrically converts a light beam in a region corresponding to the backprojected image AP1b. Thus, the photoelectric conversion units 1011a and 1011b receive light beams having parallax with each other.

[Pixel structure and sensor pupil distance]
Next, the relationship between the structure of the pixel unit 1012 and the sensor pupil distance PS1 will be described. In the same figure (a), when the x coordinate of the pixel unit 1012 is X1, the similarity between two triangles having the principal point of the microlens 1012c and the point CS1 on the sensor pupil plane as vertices,
X1 / (PS1 + h1) = dx1 / h1 (Formula 1)
It becomes. Here, since PS1 >> h1, the denominator PS1 + h1 in the left term can be approximated to PS1.
PS1 = X1 × (h1 / dx1) (Formula 2)
dx1 = h1 × (X1 / PS1) (Formula 3)
Is obtained.

  The pixel structure of the pixel portion 1012 and the dimensions of each portion are the same as those of the pixel portion 1011 except for the amount of eccentricity of the microlens. Therefore, a pair of backprojected images corresponding to the pair of photoelectric conversion units 1012a and 1012b is formed on the sensor pupil plane SPL1, and these backprojected images are backprojected images AP1a and AP1b of the photoelectric conversion unit of the pixel unit 1011. It becomes substantially the same. That is, by setting the amount of eccentricity of the microlens of each pixel so that Equation 3 is satisfied in an arbitrary pixel portion, the sensor pupil distance PS of all the pixels becomes the sensor pupil distance PS1. On the sensor pupil plane, the focus detection pupils of all the pixels are shared by AP1a and AP1b. That is, the photoelectric conversion unit included in each pixel and the sensor pupil plane are in a conjugate relationship via the microlens array, and the focus detection pupils of all the pixels substantially coincide on the sensor pupil plane.

[Baseline length of focus detection pupil (image sensor 101)]
Next, the baseline length of the focus detection pupil will be described. The pair of photoelectric conversion units 1011a and 1011b in the pixel unit 1011 has a symmetrical shape in the x direction, and the interval in the x direction of the sensitivity centroid position in each photoelectric conversion unit is GS1. Here, a line extending between the sensitivity gravity center position of the photoelectric conversion unit and the principal point 1011p of the microlens 1011c and extending in the exit pupil direction of the photographing lens is defined as a principal ray of the photoelectric conversion unit. From the similarity of two triangles formed by a pair of principal rays S11a and S11b with the principal point of the microlens as the apex,
GP1 / PS1 = GS1 / h1 (Formula 4)
Is derived, the base length GP1 of the focus detection pupil is expressed by the following equation:
GP1 = GS1 × (PS1 / h1) (Formula 5)
It becomes. In the region where the angle formed by the pair of principal rays S11a and S11b in the pair of focus detection light beams is the base line angle α11 and the approximation of sin α≈tan α≈α holds.
α11 = GS1 / h1 = GP1 / PS1 (Formula 6)
It becomes.

  Also in the pixel unit 1012 located at the end of the imaging surface, the x-direction interval between the sensitivity gravity center position of the photoelectric conversion unit 1012a and the sensitivity gravity center position of the photoelectric conversion unit 1012b is GS1, and S12b can be defined, and Equations 1 and 2 described above hold. That is, in the pixel portion 1011 and the pixel portion 1012, the baseline lengths of the focus detection pupils are equal to each other GP1. On the other hand, the baseline angles of the two are strictly different, the baseline angle of the pixel portion 1011 is α11, the baseline angle of the pixel portion 1012 is α12, and α11> α12.

  The term “base line length” refers to the distance between the entrance pupils of a pair of optical systems in a binocular stereo camera or an external measurement type phase difference detection module. In the imaging plane phase difference detection method in this embodiment, the sensor pupil plane SPL1. A distance GP1 between the upper pair of light beams is referred to as a base line length. In this case, if the sensor pupil distance PS1, which is the distance between the sensor pupil plane SPL1 and the imaging plane IP1, changes, the baseline length GP1 also changes. Therefore, when comparing the lengths of the baseline lengths, the sensor pupil distance PS1 is aligned and compared. There is a need to. In addition, although the baseline lengths GP1 and GP2 shown in FIG. 4 are the baseline lengths of the pixel unit alone, the focus detection pupil is positioned at the exit pupil of the photographing lens 500 at the time of focus detection. . Note that the baseline angle α11 does not depend on the lens pupil distance PL or the sensor pupil distance PS, but is determined only by the structure and dimensions of the pixels. Therefore, when comparing the focus detection capabilities of the image pickup elements alone, the baseline angle α11 should be compared. Can do. Since the relative lateral shift amount of the pair of two images at the time of detecting the phase difference is proportional to the baseline length GP1 or the baseline angle α11, the larger these values are, the higher the focus detection resolution is.

[Baseline length of focus detection pupil (image sensor 102)]
Next, the baseline length of the focus detection pupil corresponding to the image sensor 102 will be described. The pair of photoelectric conversion units 1021a and 1012b in the pixel unit 1021 has a symmetrical shape in the x direction like the pixel unit 1011 of the imaging element 101, and the x direction interval of the sensitivity centroid position in each photoelectric conversion unit is GS2. To do. Further, the height of the pixel portion is h2. Then, the above-described Expressions 4 to 6 in the image sensor 101 are as follows.
GP2 / PS2 = GS2 / h2 (Formula 12)
GP2 = GS2 × (PS2 / h2) (Formula 13)
α21 = GS2 / h2 = GP2 / PS2 (Formula 14)
It becomes. Since h1 = h2 and GS1 = GS2, the image sensor 101 and the image sensor 102 have the same base line angle, and α11 = α21. On the other hand, the baseline length of the focus detection pupil is GP2> GP1, and both are different values, which is due to the difference in the distance between the surface on which the focus detection pupil is projected and the imaging surface. Since the focus detection accuracy depends on the baseline angle α, the focus detection performance of the pixel unit alone at the center of the imaging surface is the same in the image sensor 101 and the image sensor 102. Since Expression 12 and Expression 13 also hold for the pixel portion 1022 at the end of the imaging surface, the focus detection performances of the imaging element 101 and the imaging element 102 are equal for the imaging element alone.

  As described above, the image pickup device 101 and the image pickup device 102 have the same basic characteristics of the respective pixel portions, but have different sensor pupil distances PS. Therefore, both have the same focus detection characteristics at the center of the imaging surface, but the positional relationship between the exit pupil of the photographing lens 500 and the focus detection pupil of the imaging device is different at the edge of the imaging surface, and the focus detection characteristics are also different (FIG. 5). To be described later).

[Focus detection principle]
The principle of focus detection using the image sensor 101 and the image sensor 102 will be described below. One photoelectric conversion unit (1011a and 1012a in FIG. 4) in the pixel unit having an arbitrary image height arranged in the image sensor 101 receives the light beam that has passed through the focus detection pupil AP1a via the microlens array, and receives a photoelectric conversion signal. Sa is output. Similarly, the other photoelectric conversion unit (1011b and 1012b in FIG. 4) in the pixel unit receives the light beam that has passed through the focus detection pupil AP1b via the microlens array, and outputs a photoelectric conversion signal Sb. Thus, signals Sa output from a plurality of pixel portions arranged continuously in the x direction and signals obtained by connecting the signals Sb are defined as an A image signal and a B image signal. Then, the A image signal and the B image signal are laterally shifted in the x direction (has parallax) in accordance with the out-of-focus state of the subject image. This lateral shift amount (also referred to as phase difference or image shift amount) is proportional to the focus shift amount of the subject image, that is, the defocus amount, and is also proportional to the baseline length GP1 or the baseline angle α11 of the focus detection pupil.

  Next, the pixel portion structure and focus detection pupil of the image sensor 102 shown in FIG. 4B will be described. In the present embodiment, the two imaging elements share one imaging lens 500, but the light beam traveling from the imaging lens 500 to the imaging element 102 is bent 90 degrees via the beam splitter 103. Therefore, the subject image formed on the image sensor 102 is a mirror image, but here, a description will be given in a state where the light beam is developed linearly and the mirror image is also returned to the original normal image.

  FIG. 4B illustrates two pixel portions 1021 and 1022 among a plurality of pixel portions included in the image sensor 102. The pixel unit 1021 is arranged at the center of the imaging surface, that is, at an image height x = 0 mm, and the pixel unit 1022 is arranged at an image height x = 10 mm.

  The pixel unit 1021 at the center of the imaging surface has the same configuration as the pixel unit 1011 in FIG. That is, the height h2 of the pixel portion of the pixel portion 1021 is the same as the height h1 of the pixel portion of the pixel portion 1011 and the surface that passes through the principal point 1021p of the microlens 1021c and is orthogonal to the optical axis S21c is the imaging surface IP2. . The optical axis of the microlens 1021c coincides with the boundary between the pair of photoelectric conversion units, and the focal position of the microlens 1021c substantially coincides with the upper surfaces of the pair of photoelectric conversion units.

  On the other hand, in the pixel portion 1022 located in the vicinity of the end portion of the imaging surface IP2, the optical axis of the micro lens 1022c is decentered with respect to the center of the pixel portion, like the pixel portion 1012 of the imaging element 101. The amount of eccentricity is set to a predetermined amount dx2 different from the aforementioned predetermined amount dx1. Therefore, the principal ray S22c of the pixel unit 1022 (a line connecting the boundary between the pair of photoelectric conversion units and the principal point of the microlens) is inclined by a predetermined angle ω2 with respect to the optical axis, and is only a predetermined distance from the imaging surface IP1. Crosses the optical axis at a separated point CS2. A virtual plane that passes through this intersection CS2 and is orthogonal to the optical axis is the sensor pupil plane SPL2 in the same manner as described with reference to FIG. 5A, and the distance PS2 between the sensor pupil plane SPL2 and the imaging plane IP2 is the sensor pupil distance PS2. Called.

Therefore, Expressions 1 to 3 related to the sensor pupil distance PS described in the image sensor 101 are as follows.
X1 / (PS2 + h2) = dx2 / h2 (Formula 7)
PS2 = X1 × (h2 / dx2) (Formula 8)
dx2 = h2 × (X1 / PS2) (Formula 9)
It becomes. The pixel structure of the pixel portion 1022 and the dimensions of each portion are the same as those of the pixel portion 1021 except for the amount of eccentricity of the microlens. Therefore, a pair of backprojected images corresponding to the pair of photoelectric conversion units 1022a and 1022b is formed on the sensor pupil plane SPL2, and these backprojected images are backprojected images AP2a and AP2b of the photoelectric conversion unit of the pixel unit 1021. It becomes substantially the same. That is, by setting the amount of eccentricity of the microlens of the pixel unit so that Equation 9 is satisfied in an arbitrary pixel unit, the sensor pupil distance PS of all the pixel units becomes the sensor pupil distance PS2. On the sensor pupil plane, the focus detection pupils of all the pixel portions are shared by AP2a and AP2b.

Here, the amount of eccentricity dx2 of the micro lens 1022c of the pixel portion 1012 is smaller than the amount of eccentricity dx1 in the image sensor 101 shown in FIG. Therefore, the magnitude relationship between the principal ray angle ω and the sensor pupil distance PS at the edge of the imaging surface is
ω1> ω2 (Formula 10)
PS1 <PS2 (Formula 11)
It becomes.

  Here, as described above, the exit pupil distance EPL1 changes depending on the zooming or focusing state change of the photographing lens 500. Furthermore, the range of change in the exit pupil distance varies depending on the type of photographic lens to be mounted. Accordingly, the sensor pupil distance PS1 and the sensor pupil distance PS2 are preferably set in consideration of the lens pupil distance EPL1 of various photographing lenses that may be attached. An example of a specific setting method will be described below. With respect to the exit pupil distance of the photographing lens 500 attached to the camera body 100, if the minimum value is PLmin and the maximum value is PLmax, the sensor pupil distances PS1 and PS2 are both set between PLmin and PLmax. If the intermediate value of the exit pupil distance is PLmid, the sensor pupil distances PS1 and PS2 are preferably set so that one of them is smaller than PLmid and the other is larger than PLmid.

  In the manufacturing process of the imaging device, a predetermined manufacturing variation occurs in the height of the pixel portion. Therefore, the principal ray angle ω and the sensor pupil distance PS also vary. Therefore, it is also important that the magnitude relationship of Equation 10 and Equation 11 does not reverse even if manufacturing variations in the height of the pixel portion occur.

  That is, it is preferable to set the eccentric amounts dx1 and dx2 of the microlens in consideration of the above conditions.

[Vapor eclipse of focus detection pupil and exit pupil according to image height]
FIG. 5 is a diagram showing a relative positional relationship between the focus detection pupil of each pixel unit and the exit pupil of the photographing lens 500 on the sensor pupil planes SPL1 and SPL2 shown in FIG. 3A is a diagram related to the pixel portion 1011 of the image sensor 101, FIG. 1B is a diagram related to the pixel portion 1012 of the image sensor 101, FIG. 1C is a diagram related to the pixel portion 1021 of the image sensor 102, and FIG. FIG. 4D is a diagram related to the pixel portion 1022 of the image sensor 102. Previously, it has been described that the size of the focus detection pupil is proportional to the projection plane distance of the focus detection pupil. Since the sensor pupil distance PS, which is the projection plane, is different between FIGS. 10A and 10B, the scale of the drawing is adjusted so that the projection distances are the same. In any figure, the optical axis position on the sensor pupil plane is the origin. First, FIG. 1A will be described.

  FIG. 5A shows the focus detection pupil and the exit pupil of the photographic lens 500 when the sensor pupil plane SPL1 is viewed from the pixel unit 1011 located at the center of the imaging plane of the imaging element 101. FIG. The origin is the intersection of the sensor pupil plane SPL1 and the optical axis, and is the point CS1 in FIG. The pair of focus detection pupils AP1a and AP1b are optically conjugate with the pair of photoelectric conversion units 1011a and 1011b included in the pixel unit 1011 via the micro lens 1011c. However, the contours of the pair of focus detection pupils AP1a and AP1b are blurred due to the optical aberration of the microlens 1011c and the diffraction of the light wave due to the small size of the pixel portion. Each focus detection pupil has higher efficiency as it goes to the center indicated by cross-hatching, in other words, the light reception intensity becomes higher as it approaches the center. Further, the efficiency of the region indicated by the halftone dots is lower toward the periphery, in other words, the received light intensity is lower. In the peripheral portion, a part of each region overlaps.

  The exit pupil EP11 is an exit pupil at the photographing lens 500 when the aperture is opened, that is, the exit pupil at F2, and the exit pupil EP11s is an aperture formed by the iris diaphragm at the time of a small aperture and has an F number larger than F2 (for example, F5.6). The exit pupil is shown in FIG. At the center of the imaging surface, the focus detection pupil and the exit pupil are symmetrical in the pupil division direction (horizontal axis u direction) with respect to the optical axis as the origin. Therefore, regardless of the F number at the time of focus detection, the shift of the pair of focus detection pupils is symmetric in the pupil division direction, and the received light signal intensity of the pair of photoelectric conversion units 1011a and 1011b included in the pixel unit 1011 is equal.

  FIG. 5B shows the focus detection pupil and the exit pupil of the photographing lens 500 when the sensor pupil plane SPL1 is viewed from the pixel unit 1012 located at the end of the imaging surface of the imaging element 101. FIG. The origin is the point CS1 in FIG. 4A, as in FIG. The shape and position of the pair of focus detection pupils AP1a and AP1b are the same as those of the pixel portion 1011 shown in FIG. 4A, and are symmetrical with respect to the origin CS1 in the pupil division direction.

  The exit pupil EP12 is an exit pupil when the aperture is opened, but the exit pupil viewed from the pixel portion 1012 has a shape surrounded by two arcs due to vignetting, and the width in the u-axis direction is narrow. Further, the exit pupil EP12s is an exit pupil at the aperture value F5.6 as in FIG. As described with reference to FIG. 4A, since the lens pupil distance PL1 and the sensor pupil distance PS1 of the first image sensor 101 are slightly different, the exit pupil center CD1 at the time of opening and narrowing is the boundary center of the focus detection pupil. It is slightly eccentric to the left with respect to CS1. However, since the amount of eccentricity is small, the displacement of the pair of focus detection pupils is substantially symmetric in the pupil division direction, and the light reception signal intensities of the pair of photoelectric conversion units 1012a and 1012b included in the pixel unit 1012 are substantially equal.

  FIG. 5C shows the focus detection pupil and the exit pupil of the photographing lens 500 when the sensor pupil plane SPL2 is viewed from the pixel unit 1021 located at the center of the imaging plane of the imaging element 102. FIG. The origin is the intersection of the sensor pupil plane SPL2 and the optical axis, and is the point CS2 in FIG. The pair of focus detection pupils AP2a and AP2b is substantially the same as that of the pixel portion 1011 in FIG.

  Further, the exit pupil EP21 when the aperture is opened and the exit pupil EP21s at F5.6 are the same as the exit pupil EP11 and the exit pupil EP11s shown in FIG. Therefore, in the pixel unit 1021, regardless of the F number at the time of focus detection, the shift of the pair of focus detection pupils is symmetric in the pupil division direction, and the received light signal intensities of the pair of photoelectric conversion units 1021 a and 1021 b are equal.

  FIG. 5D shows the focus detection pupil and the exit pupil of the photographing lens when the sensor pupil plane SPL2 is viewed from the pixel unit 1022 located at the end of the imaging surface of the imaging element 102. FIG. The origin CS2 in this figure is the point CS2 in FIG. 4B, as in FIG. The pair of focus detection pupils AP2a and AP2b are symmetric in the pupil division direction with respect to the origin CS2, as in FIG.

  The exit pupil EP22 is an exit pupil when the aperture is opened, and has a shape surrounded by two arcs by vignetting as in FIG. Further, the exit pupil EP22s is an exit pupil at the aperture value F5.6 as in the other drawings. As described with reference to FIG. 4B, since the lens pupil distance PL1 and the sensor pupil distance PS2 of the image sensor 102 are considerably different, the exit pupil center CD2 at the time of opening and narrowing is relative to the boundary center CS2 of the focus detection pupil. Greatly deviates to the left. Therefore, the shift of the pair of focus detection pupils is asymmetric in the pupil division direction, and the received light signal strengths of the pair of photoelectric conversion units 1022a and 1022b included in the pixel unit 1022 are greatly different. This light amount difference (strictly speaking, the light amount ratio) becomes more significant as the image height increases and the aperture becomes smaller.

[Shading characteristics and focus detection performance]
FIG. 6 shows the amount of light received by the image sensor, that is, the shading characteristics, when an object having a uniform luminance distribution is captured when the aperture value of the photographic lens 500 is widened (F2 in this case) and narrowed down to F5.6. It is a figure explaining. 3A is a characteristic diagram of the image sensor 101, and FIG. 1B is a characteristic diagram of the image sensor 102. FIG. First, FIG. 1A will be described.

  The horizontal axis of FIG. 6A is the image height in the x direction on the imaging surface IP1 of the image sensor 101, and the vertical axis is the output signal of the photoelectric conversion unit that receives the light beam corresponding to the pair of focus detection pupils. It is a value proportional to the amount of light received by the photoelectric conversion unit provided. S1a (F2) is an output signal of the photoelectric conversion unit group corresponding to one focus detection pupil AP1a at the time of full aperture (F2), and this is referred to as an A image signal. S1b (F2) is an output signal of the photoelectric conversion unit corresponding to the other focus detection pupil AP1b to be paired, and is referred to as a B image signal. At the time of focus detection, a defocus amount is detected by performing a correlation operation between the A image signal and the B image signal for a subject having a predetermined light and dark pattern. On the other hand, the waveform in the figure represents the signal characteristic for a solid uniform luminance surface, that is, the shading characteristic of the focus detection system. According to the figure, the values of both signals rapidly decrease as the image height x increases. This is due to the vignetting of the taking lens. In the imaging device 101, the lens pupil distance PL1 and the sensor pupil distance PS1 substantially coincide with each other, but strictly speaking, the lens pupil distance PL1 is slightly smaller as described with reference to FIG. Therefore, as described with reference to FIGS. 5A and 5B, the center of the exit pupil of the photographic lens and the neutral point of the focus detection pupil are slightly shifted as the image height increases, so that the A image signal in FIG. The values of S1a (F2) and B image signal S1b (F2) are slightly different depending on the image height. Further, in the region where the image height x is positive, A image> B image, and in the region where the image height x is negative, A image <B image, and the magnitude relationship of both signals is reversed depending on whether the image height is positive or negative.

  S1a (F5.6) and S1b (F5.6) in FIG. 6A are A and B image signals when the aperture is reduced to F5.6. The output of both signals gradually decreases with the image height, and the cause of the decrease is due to the cosine fourth law. Both signals have different values as the image height increases. The reason is the same as in the case of full aperture.

  FIG. 6B is a characteristic diagram of the image sensor 102 and corresponds to FIG. S2a (F2) and S2b (F2) are the A and B image signals when the aperture is open (F2), and S2a (F5.6) and S2b (F5.6) are the A values when the aperture is reduced to F5.6. An image signal and a B image signal. As described with reference to FIG. 4B, the image sensor 102 has a larger discrepancy between the sensor pupil distance PS <b> 2 and the current lens pupil distance PL <b> 1 than the image sensor 101. For this reason, as shown in FIG. 5D, in the region where the image height is high, the deviation between the exit pupil center of the photographing lens and the neutral point of the focus detection pupil is large, and the light quantity ratio between the pair of focus detection pupils is also unbalanced. . Therefore, the shading characteristic of the image sensor 102 is also caused by the difference between the A image signal and the B image signal caused by the positional deviation between the exit pupil and the focus detection pupil, in addition to the light amount decrease caused by vignetting and the cosine fourth law. It is larger than the element 101. If the correlation calculation is performed while the difference between the A image signal and the B image signal is large, the accuracy of focus detection is reduced. For this reason, at the time of focus detection, inter-phase calculation is generally performed after shading the A image signal and the B image signal using shading correction information stored in the imaging apparatus. However, if the level difference between the original A image signal and the B image signal is large, the shading correction error tends to be large. If the original signal level is lower than the originally assumed level, the signal amplification ratio by shading correction also increases, and as a result, noise and the like are amplified, and the focus detection accuracy decreases. Comparing the shading characteristics of FIGS. 6A and 6B based on the contents described above, the level difference between the A image signal and the B image signal is smaller in FIG. 6A. It can be said that the smaller the level difference between the A image signal and the B image signal, the higher the focus detection accuracy. Therefore, when both of the two image pickup devices can detect the phase difference, the image pickup device having the smaller deviation between the lens pupil distance PL and the sensor pupil distance PS and having the same shading characteristics of the A image signal and the B image signal. It is preferable to preferentially use the focus detection result.

The degree of coincidence of the shading characteristics is determined by defining as follows. At a predetermined image height close to the edge of the imaging surface, the larger one of the A image signal and the B image signal is Max (A, B), the smaller one is Min (A, B), and the ratio of both is the shading ratio SH. The following equation SH = Max (A, B) / Min (A, B) (Equation 14)
Define in. FIG. 7 is a diagram showing how the shading ratio SH in the two image sensors changes with respect to the lens pupil distance PL. The horizontal axis represents the lens pupil distance PL of the photographing lens, and the vertical axis represents the shading ratio SH. As described above, in the photographing lens 500 of the present embodiment, the lens pupil distance PL changes from PLmin to PLmax due to a change in zoom or focus state. The shading ratios SH1 (F2) and SH1 (F5.6) are the shading ratios of the image sensor 101 when the aperture value of the photographing lens is full and F5.6. When the lens pupil distance PL1 coincides with the sensor pupil distance PS1 of the image sensor 101, the shading waveforms of the A image signal and the B image signal are equal, and the shading ratio SH1 becomes a minimum value of 1. As the difference between the lens pupil distance PL1 and the sensor pupil distance PS1 increases, the shading ratio SH1 increases, and the shading ratio increases significantly as the aperture of the photographing lens 500 becomes smaller.

  Similarly, the shading ratios SH2 (F2) and SH2 (F5.6) are shading ratios of the image sensor 102 when the aperture value of the photographing lens is opened and reduced to F5.6. When the lens pupil distance PL1 coincides with the sensor pupil distance PS2 of the image sensor 102, the shading waveforms of the A image signal and the B image signal are equal, and the shading ratio SH1 has a minimum value of 1. As the lens pupil distance PL1 deviates from the sensor pupil distance PS2, the shading ratio SH2 increases, and the shading ratio increases significantly as the aperture of the taking lens 500 becomes smaller.

  The boundary pupil distance PSmid (first predetermined distance) is a lens pupil distance PL at which the magnitude relationship between the shading ratios SH1 and SH2 is reversed. The boundary pupil distance PSmid is larger than the sensor pupil distance PS1 of the image sensor 101 and smaller than the sensor pupil distance PS2 of the image sensor 102. When the lens pupil distance is equal to or less than the boundary pupil distance PSmid (first predetermined distance), the shading ratio SH1 of the image sensor 101 is smaller than the shading ratio SH2 of the image sensor 102. When the lens pupil distance is larger than the boundary pupil distance PSmid (first predetermined distance), the shading ratio SH2 of the image sensor 102 is smaller than the shading ratio SH1 of the image sensor 101. Since the level difference between the A image signal and the B image signal is smaller as the shading ratio SH is smaller, it is preferable to use an image sensor having a smaller shading ratio SH for focus detection. Therefore, in this embodiment, when the lens pupil distance PL1 at the time of focus detection is equal to or less than the boundary pupil distance PSmid (less than the first predetermined distance), priority is given to the focus detection result of the image sensor 101. When the lens pupil distance PL1 is longer than the boundary pupil distance PSmid (predetermined distance), priority is given to the focus detection result of the image sensor 102. Details will be described later.

  FIG. 8 is an example of a waveform of a focus detection signal when a subject existing within a predetermined focus detection area is detected. FIG. 8A shows a waveform before shading correction, and FIG. 8B shows a waveform after shading correction. is there. In both figures, the horizontal axis is the x coordinate of the imaging surface, the vertical axis is the output intensity of the focus detection signal, Sa is the A image signal output from the image sensor 101 or 102, and Sb is the B image signal. Since FIG. 5A is before shading correction, a level difference is generated between the pair of signals. That is, the average value of the A image signal Sa is higher than that of the B image signal Sb, and the level difference increases as the image height x increases.

  FIG. 4B shows a focus detection signal after a shading correction process described later, in which the shading of the A image signal Sa and the B image signal Sb is eliminated and the signal levels are uniform. Therefore, the phase difference φ between the two images is calculated by a predetermined interphase calculation in the signal of the pair of two images after the shading correction.

  Various expressions have been proposed as interphase arithmetic expressions. For example, the following expression 15 is used.

      C (φ) = Σ | A (i) −B (i) | (Formula 15)

  A (i) and B (i) are an A image signal and a B image signal output from a predetermined focus detection area, and i represents a pixel number in the x-axis direction. For example, if the number of focus detection pixels existing in the x-axis direction of the focus detection area is 100 pixels, i takes a value from 1 to 100. Therefore, the right side of Expression 15 is obtained by integrating the absolute values of the differences between the A image signal and the B image signal in the focus detection area. C (φ) is a correlation value, and φ is a relative shift amount between the A image signal and the B image signal when performing the integration calculation. That is, the interphase calculation calculates Equation 15 while relatively shifting the A image signal and the B image signal, and φ when the correlation value C (φ) takes the minimum value is the phase difference between the A image signal and the B image signal. I reckon.

[Acquisition of phase difference and calculation of defocus amount (focus detection)]
FIG. 9 is a diagram illustrating the relationship between the phase difference between the pair of two images and the correlation value. The horizontal axis represents the relative shift amount of the A image signal and the B image signal, and the vertical axis represents the correlation value C (φ) of Equation 15. C1 is a correlation value of the focus detection signal of the image sensor 101, and the correlation value indicates a minimum value C1min at the shift amount φ1. C2 is a correlation value of the focus detection signal of the image sensor 102, and the correlation value indicates a minimum value C2min at the shift amount φ2. In FIG. 4, since the degree of coincidence between the lens pupil distance PL and the sensor pupil distance PS is better for the image sensor 101, the signal output from the image sensor 101 is often more reliable as the focus detection signal. Therefore, it is estimated that the minimum value of the correlation value C (φ) is smaller by the image sensor 101, and the obtained phase difference is more reliable in φ1 than in φ2.

  Next, the phase difference φ obtained by Expression 15 is converted into a defocus amount DEF. Here, the phase difference φ and the defocus amount DEF are in the relationship of the following equation.

φ = DEF × α (Formula 16)
DEF = φ / α = φ × K (Equation 17)

  α is a baseline angle of a pair of focus detection light beams, which is α11 and α12 in FIG. 4A or α21 and α22 in FIG. However, although the base line angle shown in FIG. 4 is a value in the image pickup device alone, the focus detection light beam is injured corresponding to the exit pupil of the photographing lens 500. Accordingly, the baseline angle α in Expression 16 and Expression 17 also changes depending on the optical state of the photographing lens 500, that is, the F number of the photographing lens 500 and the lens pupil distance PL. Therefore, in order to calculate the defocus amount DEF, information on the baseline angle α is necessary. Details of the baseline angle α will be described later.

  The camera CPU 104 converts the phase difference φ into the defocus amount DEF using Expression 17, and further converts it into the drive amount of the focus lens and transmits it to the lens CPU 507 of the photographing lens 500. Then, the lens CPU 507 controls the focus lens actuator 506 based on the received lens driving amount to focus the subject image.

[Shading correction information]
FIG. 10 shows shading correction information included in the image pickup apparatus of the present embodiment, which is stored in a memory such as the storage medium 106 as a lookup table. FIG. 4A shows information stored for the image sensor 101, and FIG. 4B shows information for the image sensor 102. FIG. In FIG. 6A, in the row direction (lateral direction) of the lookup table, the F number of the taking lens 500 is assigned as eight values for each aperture from F1.4 to F16. In the row direction (vertical direction), the lens pupil distance PL of the photographic lens 500 is an equal number sequence or a ratio number sequence from a minimum lens pupil distance 1 (for example, 50 mm) to a maximum lens pupil distance 8 (for example, 200 mm). Assigned as. Then, shading correction information Fs111 to Fs188 corresponding to the image sensor 101 is stored at locations corresponding to the respective F numbers and lens pupil distances. Here, Fs111 to Fs188 are not a single constant, but are composed of a plurality of coefficients for defining a predetermined function. As described with reference to FIG. 6, the shading generated in the focus detection signal continuously changes according to the image height on the imaging surface. Therefore, the shading correction function may be defined by a polynomial function having the image height x and the image height y as variables, and the coefficients in each order when the shading waveform in FIG. 6 is approximated by the function may be stored.

  FIG. 6B shows shading correction information for the image sensor 102, and the content thereof is the same as that shown in FIG.

[Conversion information for converting phase difference φ between two images into defocus amount]
FIG. 11 shows conversion information for converting the phase difference φ between the two images shown in FIG. 8B into a defocus amount, which corresponds to K in Expression 17 and is stored as, for example, a lookup table similar to FIG. It is stored in a memory such as medium 106. 11A shows the conversion information stored for the image sensor 101, and FIG. 11B shows the conversion information for the image sensor 102. In FIG. 9A, conversion information Fk111 to Fk188 corresponding to the image sensor 101 is stored at locations corresponding to the F number and lens pupil distance in the lookup table. Here, Fk111 to Fk188 are not a single constant, but are composed of a plurality of coefficients for defining a predetermined function. Since the focus detection light beam received by each pixel unit on the imaging surface at the time of phase difference detection is changed according to the x and y coordinates on the imaging surface, the base length of the focus detection pupil is also continuous according to the image height. Changes. Therefore, the coefficient for converting the phase difference φ into the defocus amount is also defined by a polynomial function having the image height x and the image height y as variables, and the conversion coefficient distribution obtained by optical calculation or actual measurement is approximated by the function. What is necessary is just to memorize | store the coefficient in each order of the performed function.

  FIG. 6B shows conversion information for the image sensor 102, and the content thereof is the same as that shown in FIG.

[Main Flowchart of First Embodiment (FIG. 12)]
FIG. 12 is a main flowchart showing the procedure of photographing processing in this embodiment. When the photographer turns on the power switch of the camera in S101, the camera CPU 104 checks the operation of each actuator in the camera, the image sensor 101, and the image sensor 102, and initializes the memory contents and the execution program.

  In S102, the camera CPU 104 communicates with the lens CPU 507 and receives information such as the open F number of the photographing lens 500, the focal length, the lens pupil distance PL, and the focus sensitivity that is a proportional constant between the focus lens extension amount and the focus change amount. .

[When shooting still images]
In S103, the camera CPU 104 determines whether the shooting mode is the still image mode or the moving image mode. If the shooting mode is the still image mode, the process proceeds to S111. If the shooting mode is the moving image mode, the process proceeds to S131.

  In step S111, the camera CPU 104 controls to drive the image sensor 101 for still image shooting in the live view mode. The live view is a mode in which an image acquired by the image sensor is displayed on the in-finder display 107 or the external display 110 in FIG. 1 in real time. Since the number of pixels of the display is small in both the horizontal and vertical directions with respect to the number of pixels of the image for recording, the pixels are thinned out in both the horizontal and vertical directions when reading from the image sensor in the live view mode. The power consumption of the signal processing circuit is kept low. The camera CPU 104 (first focus detection means) also performs phase difference detection using the image signal read out in the live view mode. However, in order to maintain the resolution of the focus detection signal, only the focus detection area is not thinned out and read out. Information on the pixel portion may be read out.

  In S112, the camera CPU 104 determines the brightness of the image signal acquired in S111, and performs aperture control during live view. Steps S111 to S115 are steps for performing live view and focus adjustment during still image shooting. However, there is no significant problem even when the aperture value is different between live view and still image shooting during still image shooting. On the other hand, a still image has a large number of recorded pixels and a high resolution of the image, so that a tolerance value for focusing error is small. Therefore, in S112, the camera CPU 104 performs aperture control via the lens CPU 507 so that the F number of the photographing lens 500 is small, that is, the aperture aperture diameter is large. Specifically, the F number of the photographing lens is set to the F number closer to the open position, and the determined F number information is transmitted to the lens CPU 507 via the camera side communication terminal 113 and the lens side communication terminal 508. Then, the lens CPU 507 controls the aperture driving unit 506 to control the aperture diameter of the iris diaphragm to a predetermined value. When the amount of light passing through the photographic lens 500 is large and overexposed, the amplifier gain for amplifying the signal of the image sensor is lowered and the accumulation time of the electronic shutter for controlling the exposure time is shortened.

  In S113, the signal acquired by the image sensor 101 is converted into a display signal and transmitted to the in-finder display 107 or the external display 110 to start live view display.

  In step S114, the camera CPU 104 controls to execute a focus detection 1 subroutine suitable for still image shooting. Details of this subroutine will be described with reference to FIG.

  In S115, the camera CPU 104 transmits the focus lens driving amount calculated in S114 to the lens CPU 507 via the camera side communication unit 113 and the lens side communication unit 508. Then, the lens CPU 507 controls to drive the focus driving unit 504 by a predetermined amount in order to eliminate the focus shift.

  In S116, the camera CPU 104 determines whether or not the still image start trigger button has been turned on. If it is not turned on, the process returns to S111, and the live view display and the focus adjustment operation of S111 to S115 are repeatedly executed. If the still image start trigger button has been turned on, the process proceeds from S116 to S121.

  In S121, the camera CPU 104 controls to execute the still image shooting 1 subroutine. Details of this subroutine will be described with reference to FIG.

  In S122, the camera CPU 104 processes the signal acquired in S121, generates a still image signal, and records it in the storage medium 106 (memory means). Since each pixel unit of the image sensor 101 has a pair of two photoelectric conversion units for phase difference detection, the output signal is also composed of a pair of two signals for each pixel unit. Therefore, the AFE 114 (addition control means) adds a pair of signals for each pixel unit to obtain an image signal for recording or display of each pixel unit. Next, processing such as color conversion for demosaicing the Bayer array color information, gamma correction, and compression is performed to generate a recording image. Note that processing of the signal and generation of a recorded image may be performed by providing a dedicated circuit.

  In step S123, the camera CPU 104 performs control so as to end the still image shooting.

[When recording movies]
Next, a flow at the time of moving image shooting will be described. In S103, the camera CPU 104 determines whether or not the moving image shooting mode is set. If the moving image shooting mode is set, the process proceeds from S103 to S131.

  In S131, the camera CPU 104 controls to drive the image sensor 102 for moving image shooting in the live view mode.

  In S <b> 132, the camera CPU 104 performs aperture control for moving image recording via the aperture drive unit 506. During movie shooting, the aperture for live view and movie recording is the same, so in this step, an aperture value suitable for the movie is selected. If the exposure time of the electronic shutter is too short during moving image shooting, the continuity of a moving subject is lost, resulting in an unnatural moving image in which a still image of stop motion is frame-fed at high speed. Therefore, the time of the electronic shutter that avoids such a phenomenon is selected, and the aperture value of the photographing lens and the amplifier gain of the image sensor are appropriately controlled so as to achieve proper exposure. The F number determined here is transmitted by the camera CPU 104 to the lens CPU 507 via the camera side communication terminal 113 and the lens side communication terminal 508. Then, the lens CPU 507 controls the aperture driving unit 506 to control the aperture diameter of the iris diaphragm 505 to a predetermined value.

  In S133, the camera CPU 104 converts the signal acquired by the image sensor 102 into a display signal, and transmits it to the in-finder display 107 or the external display 110 to control to start live view display.

  In S134, the camera CPU 104 controls to execute a focus detection 2 subroutine suitable for moving image shooting. Details of this subroutine will be described with reference to FIG.

  In S135, the camera CPU 104 transmits the focus lens driving amount calculated in S134 to the lens CPU 507 via the camera side communication terminal 113 and the lens side communication terminal 508. Then, the lens CPU 507 controls to drive the focus driving unit 504 by a predetermined amount in order to eliminate the focus shift.

  In S141, it is determined whether or not the moving image shooting trigger button is turned on. If the ON operation has been performed, the AFE 115 performs image processing for a moving image and generates a moving image in S142. When the generated moving image is recorded, the process proceeds to S143. If the movie shooting trigger button is not turned on, the procedure jumps from S141 to S143 without recording a movie.

  In S143, the camera CPU 104 determines whether or not the still image start trigger button has been turned on. In this embodiment, when a still image shooting is instructed during live view for moving images or recording of a moving image, recording of a still image by the image sensor 101 is possible. Therefore, if the still image start trigger button has been turned on, the process proceeds from S143 to S144.

  In S144, the camera CPU 104 controls to execute the still image shooting 2 subroutine. In this subroutine, unlike the still image shooting 1 subroutine of S121, the camera CPU 104 controls the still image shooting in parallel with the moving image shooting in the state where the moving image shooting mode is selected. Details will be described with reference to FIG.

  In S145, the camera CPU 104 processes the still image signal acquired in S144, generates a still image signal, and records it in the storage medium 106. Note that the processing of the signal and the generation of the recorded image may be performed in the circuit provided with a dedicated circuit. The specific processing content is the same as S122 described above.

  After executing S145, the process proceeds to S146, and the camera CPU 104 determines whether or not the moving image shooting trigger button is turned off. If the on-state continues, steps S131 to S145 are repeatedly executed to continue the AF control for moving images and the moving image recording, and also permit interruption of still images. On the other hand, when the camera CPU 104 determines that the moving image shooting trigger button has been turned off in S146, the process proceeds to S123 and the moving image shooting is ended.

[Focus Detection 1 (FIG. 13A)]
FIG. 13A is a “focus detection 1” flow executed at the time of still image shooting, and is a subroutine executed in S114 of FIG.

  When the process proceeds from S114 to S151, the camera CPU 104 compares the lens pupil distance PL of the photographing lens 500 transmitted from the lens CPU 507 with the boundary pupil distance PSmid described in FIG. If the lens pupil distance PL is equal to or less than the boundary pupil distance PSmid (first predetermined distance), the process proceeds to S152. In S152, the camera CPU 104 performs focus detection by the image sensor 101 that is driven for still image shooting. Specifically, first, the AFE 114 generates a pair of focus detection signals (A image signal and B image signal) from the signals acquired by the image sensor 101. Then, the camera CPU 104 reads information corresponding to the F number and the lens pupil distance PL at the time of focus detection from the look-up table for shading correction shown in FIG. 10A, and converts the information into an A image signal and a B image signal. Apply shading correction. Next, the camera CPU 104 calculates the phase difference φ1 between the A image signal and the B image signal by the interphase calculation using the interphase calculation formula of Formula 15. Then, the camera CPU 104 reads conversion information for calculating the defocus amount DEF from the look-up table shown in FIG. 11A, and calculates (focus detection) the defocus amount DEF using the above-described equation 17. , The process proceeds to S155. In S155, the camera CPU 104 converts the defocus amount into the focus lens drive amount, and returns to the main flow in S156.

  On the other hand, when the camera CPU 104 determines in S151 that the current lens pupil distance PL is longer than the boundary pupil distance PSmid (first predetermined distance), the process proceeds to S153.

  In S153, the camera CPU 104 drives the moving image pickup element 102 in the non-driven state, and performs the same control as S152 described above. That is, first, the AFE 115 generates a pair of focus detection signals (A image signal and B image signal) from signals acquired by the image sensor 102. Then, the camera CPU 104 reads information corresponding to the F number and the lens pupil distance PL at the time of focus detection from the look-up table for shading correction shown in FIG. 10B, and converts the information into an A image signal and a B image signal. Apply shading correction. Next, the camera CPU 104 calculates a phase difference φ2 between the A image signal and the B image signal by using a known interphase calculation algorithm. Then, the camera CPU 104 reads the conversion information for calculating the defocus amount DEF from the lookup table shown in FIG. 11B, calculates the defocus amount DEF using the above-described equation 17, and proceeds to S155. To do. In S155, the camera CPU 104 converts the defocus amount into the focus lens drive amount, and the camera CPU 104 or the lens CPU 507 returns to the main flow in S156.

[Effects of processing of focus detection 1]
The effect of the focus detection 1 flow (FIG. 13A) will be described.

  As described above, the shading ratio SH1 increases as the lens pupil distance PL1 deviates from the sensor pupil distance PS1. The fact that the shading ratio is large means that the level difference between the A image signal and the B image signal is large. Therefore, when the shading ratio is large, the focus detection accuracy is lower than when the shading ratio is small.

  Accordingly, in the focus detection 1, when the lens pupil distance PL1 is equal to or less than the boundary pupil distance PSmid (S151), the camera CPU 104 performs focus detection using the signal acquired from the image sensor 101 (S152). This is because the distance of the sensor pupil distance PS1 is closer to the current lens pupil distance PL1 than the sensor pupil distance PS2.

  Further, in the focus detection 1, when the lens pupil distance PL1 is longer than the boundary pupil distance PSmid (S151), the camera CPU 104 performs focus detection using the signal acquired from the image sensor 102 (S154). This is because the sensor pupil distance PS2 is closer to the lens pupil distance PL1 than the sensor pupil distance PS1.

  As described above, in the focus detection 1, the camera CPU 104 performs focus detection using the image sensor having the sensor pupil distance PS closer to the lens pupil distance PL1. Thereby, compared with the case where the sensor pupil distance PS of two image sensors is the same, it can respond also to the change of the lens pupil distance PL1 of a wider range. That is, it is possible to perform focus detection with better accuracy.

[Still Image Shooting 1 (FIG. 13B)]
FIG. 13B is a flow of “still image shooting 1”, which is a subroutine executed by the camera CPU 104 in S121 of FIG. After shifting from S121 to S161, the camera CPU 104 transmits an F number for still image shooting to the lens CPU 507 via the camera-side communication terminal 113 and the lens-side communication means 508. The lens CPU 507 controls the aperture driving unit 506 to control the aperture diameter of the iris diaphragm 505 to a value corresponding to an F number suitable for still image shooting.

  In S162, the camera CPU 104 controls the shutter driving unit 112 so as to reset the focal plane shutter 111 that has been opened for live view to the closed state.

  In S163, the image sensor 101 starts a charge accumulation operation for taking a still image.

  In S164, the shutter drive unit 112 drives the front curtain and the rear curtain of the focal plane shutter 111 based on the still image shooting shutter time calculated by the predetermined exposure calculation program, and gives a predetermined exposure amount to the image sensor.

  When the travel of the focal plane shutter 111 is completed, the accumulation operation of the image sensor 101 is terminated in S165, and charge transfer is performed. In S166, the process returns to the main flow.

  As described above, when still image shooting is performed in a state where the still image shooting mode is selected, the focal plane shutter 111 performs light amount adjustment, and blocks the light flux reaching the image sensor during charge transfer. As a result, occurrence of smear and blooming can be avoided, and a high-quality still image can be obtained.

[Focus Detection 2 (FIG. 14A)]
FIG. 14A is a “focus detection 2” flow executed during moving image shooting, and is a subroutine executed in S134 of FIG.

  When the process shifts from S134 to S171, the camera CPU 104 compares the lens pupil distance PL1 of the photographing lens at the present time with the boundary pupil distance PSmid described with reference to FIG. When the camera CPU 104 determines that the lens pupil distance PL1 is longer than the boundary pupil distance PSmid, the process proceeds to S172. In step S172, the camera CPU 104 performs focus detection by the image sensor 102 that is driven for moving image shooting. The content of the control is substantially the same as S154 in FIG. When the camera CPU 104 calculates the defocus amount DEF in S172, the process proceeds to S175. In S175, the camera CPU 104 converts the defocus amount into the focus lens drive amount, and returns to the main flow in S176.

  On the other hand, if the current lens pupil distance PL1 is equal to or smaller than the boundary pupil distance PSmid (first predetermined distance) in S171, the camera CPU 104 proceeds to S173.

  In S173, the camera CPU 104 controls to drive the still image sensor 101 that is in the non-driven state, and in S174, the same calculation as S152 in FIG. 13A described above is performed. When the camera CPU 104 calculates the defocus amount DEF in S174, the process proceeds to S175 to convert the defocus amount into the focus lens drive amount, and returns to the main flow in S176.

[Effects of processing of focus detection 2]
The effect of the “focus detection 2” flow (FIG. 14A) will be described.

  For the same reason as described in the focus detection 1, the focus detection accuracy of the image sensor 102 is lower when the deviation between the lens pupil distance PL1 and the sensor pupil distance PS2 is larger than when the deviation is small. To do.

  Therefore, in the focus detection 1, when the lens pupil distance PL1 is longer than the boundary pupil distance PSmid (S171), focus detection is performed using a signal acquired from the image sensor 102 (S172). This is because the sensor pupil distance PS1 is closer to the current lens pupil distance PL1 than the sensor pupil distance PS2.

  In the focus detection 2, when the lens pupil distance PL1 is equal to or less than the boundary pupil distance PSmid (S171), focus detection is performed using the signal acquired from the image sensor 102 (S173). This is because the sensor pupil distance PS1 is closer to the lens pupil distance PL1 than the sensor pupil distance PS2.

  As described above, in the focus detection 2, similarly to the focus detection 1, the detection is performed using the imaging element having the sensor pupil distance PS closer to the lens pupil distance PL1. Thereby, compared with the case where the sensor pupil distance PS of two image sensors is the same, it can respond also to the change of the lens pupil distance PL1 of a wider range. That is, it is possible to perform focus detection with better accuracy.

[Still Image Shooting 2 (FIG. 14B)]
FIG. 14B is a flow of “still image shooting 2” for recording a still image during moving image shooting, and is a subroutine executed by the camera CPU 104 in S144 of FIG.

  At the time of shifting from S144 to S181, the focal plane shutter 111 is in an open state, and the iris diaphragm 505 of the photographing lens 500 is controlled to an F number for moving image photographing. In step S181, the camera CPU 104 calculates the exposure time of the still image sensor 101 based on the F number and the sensitivity set for the image sensor 101 at this time.

  In step S182, the camera CPU 104 controls the electronic shutter of the image sensor 101 to operate while the focal plane shutter 111 is open. Thereby, the image sensor 101 starts charge accumulation.

  In S183, after a predetermined time has elapsed, the image sensor 101 closes the electronic shutter to finish charge accumulation, performs charge transfer in S184, and returns to the main flow in S184.

  As described above, when still image shooting is performed in parallel with moving image shooting in a state in which the moving image shooting mode is selected, the camera CPU 104 controls to perform still image shooting with the F number controlled in the moving image shooting mode. To do. Further, the focal plane shutter 111 is not operated, and exposure control by the electronic shutter is performed. Therefore, since the aperture diameter of the aperture does not change inadvertently during moving image recording and the operation sound of the focal plane shutter 111 is not recorded, a still image can be obtained while continuing high-quality moving image recording. Can do.

[Effects of Example 1]
In the first embodiment described above, the sensor pupil distance PS1 of the image sensor 101 and the sensor pupil distance PS2 of the second image sensor are made different. Then, as described in S151 in FIG. 13A and S171 in FIG. 14A, the magnitude relationship between the lens pupil distance PL1 and the boundary pupil distance PSmid at the time of focus detection is determined, and the sensor pupil distance PS and the lens pupil are determined. Focus detection is performed by an image sensor having a smaller distance PL. As a result, since focus detection is performed using the image sensor with a smaller level difference between the A image signal and the B image signal, highly reliable focus detection is possible, and high-quality still images and moving images that are in focus can be obtained. You can get it.

  In the first embodiment, the image sensor 101 is mainly for still image shooting, the second image sensor is mainly for moving image shooting, and the sensor pupil distance PS is longer in the image sensor 102 than in the image sensor 101. The reason is that the photographing lens 500 suitable for moving image photographing tends to be designed so that the lens pupil distance PL becomes long. When the focus lens is driven at the time of focus adjustment, not only the focus state but also a change in image size, so-called image magnification fluctuation, occurs. If the image magnification fluctuation occurs during moving image recording, the quality of the moving image is deteriorated. Therefore, the moving image shooting lens is designed to be so-called image side telecentric with a long lens pupil distance PL in order to suppress a change in image magnification. Is preferred. Therefore, in the first embodiment, in view of the above background, the sensor pupil distance PS of the imaging element 102 for moving image shooting is longer than that of the imaging element 101. However, the present invention is not limited to this. For example, when the image sensor 101 is used for moving images, the sensor pupil distance PS may be increased.

  In the first embodiment, the length of the sensor pupil distance PS is used as a criterion for selecting an image sensor used for focus detection. However, the present invention is not limited to this. For example, the shading correction information shown in FIG. 10 can be used as a determination criterion. In FIG. 10, when the F number at the time of focus detection is F2 and the lens pupil distance is lens pupil distance 2, the shading correction information in the image sensor 101 is Fs122 and the shading correction information in the image sensor 102 is the same as FIG. From (b), it is Fs222. Then, by comparing the values of Fs122 and Fs222, it is possible to estimate the level difference between the A image signal and the B image signal. That is, an embodiment in which information stored for correcting a focus detection signal is compared and an image sensor used for focus detection is selected based on the result is also possible. In this embodiment, the image pickup device that is not used for recording a still image or a moving image and is determined not to be suitable for focus detection is not driven, so unnecessary power consumption is avoided and focusing accuracy is improved. And power saving. However, the present invention is not limited to this. By simultaneously driving the two image sensors, when a focus detection signal is acquired from an image sensor that is not used for recording, the drive time can be shortened and the drive timing can be adjusted. In this case, the camera body 100 has separate image processing units corresponding to the two image sensors.

[Example 2]
In the first embodiment, information on the lens pupil distance PL and the sensor pupil distance PS and shading correction information are used as a determination criterion for selecting an image sensor used for focus detection. On the other hand, in the second embodiment described below, the camera CPU 104 performs focus detection using signals from both of the two image sensors at the time of focus detection, and a focus that is estimated to have high reliability among the focus detection results. The detection result is selected. Since the configuration and focus detection characteristics of the imaging apparatus in the second embodiment are the same as those in the first embodiment, description thereof is omitted. Although the main flow of the photographing process in the second embodiment is the same as the main flowchart shown in FIG. 12, the focus detection subroutines S114 and S132 are different, and only the focus detection subroutine will be described.

  In the main flow of FIG. 12, in the still image mode, the process proceeds from S103 to S111, S111 to S113 are executed, and the process proceeds to S114. In S114, the focus detection 3 subroutine shown in FIG. 15A is executed.

[Focus Detection 3 (FIG. 15A)]
FIG. 15A is a flowchart of “focus detection 3” executed at the time of still image shooting of the camera CPU 104 in the image pickup apparatus of the second embodiment.

  After shifting from S114 to S251, the camera CPU 104 performs focus detection (defocus amount calculation) by the image sensor 101 that is driven for still image shooting. Specifically, the camera CPU 104 controls the AFE 114 to generate a pair of focus detection signals (A image signal and B image signal) from signals acquired by the image sensor 101. Then, the camera CPU 104 reads information corresponding to the F number and the lens pupil distance PL at the time of focus detection from the look-up table for shading correction shown in FIG. 10A, and converts the information into an A image signal and a B image signal. Control to perform shading correction. Next, the camera CPU 104 calculates the phase difference φ1 between the A image signal and the B image signal and the minimum value C1 of the correlation value by the interphase calculation using the interphase calculation formula of Formula 15. Then, the camera CPU 104 reads conversion information for calculating the defocus amount DEF from the look-up table shown in FIG. 11A, and calculates the defocus amount DEF using the above-described equation 17.

  In step S252, the camera CPU 104 performs control so as to drive the moving image pickup element 102 in a non-driven state. In S253, the camera CPU 104 performs the same control as in S251 described above. That is, the camera CPU 104 controls the AFE 115 to generate a pair of focus detection signals (A image signal and B image signal) from the signal acquired by the image sensor 102. Then, the camera CPU 104 reads information corresponding to the F number and the lens pupil distance PL at the time of focus detection from the look-up table for shading correction shown in FIG. 10B, and performs shading into an A image signal and a B image signal. Control to apply correction. Next, the camera CPU 104 calculates the phase difference φ2 between the A image signal and the B image signal and the minimum value C2 of the interphase value by the interphase operation using the interphase arithmetic expression of Expression 15. Then, the camera CPU 104 reads conversion information for calculating the defocus amount DEF from the look-up table shown in FIG. 11B, calculates the defocus amount DEF using the above-described equation 17, and proceeds to S254. To do.

  In S254, the camera CPU 104 (first determination unit) compares and determines the reliability of the focus detection result obtained by the image sensor 101 obtained in S251 and the focus detection result obtained by the second image sensor obtained in S253. As an example of an index indicating reliability, the minimum value of the correlation value described in FIG. 9 is used in the present embodiment. When the level difference between the A image signal and the B image signal is large, the signal having the lower level has a larger amplification gain due to shading correction than when the level difference is small. Then, the noise component of the original signal and the output error due to the sensitivity variation of each pixel unit are enlarged, and the degree of coincidence of the two images is lowered. In this case, the phase difference φ indicating the minimum value of the interphase value also shows a relatively large value. That is, as the minimum value of the interphase value is lower, the degree of coincidence between the A image signal and the B image signal is higher, and the reliability of the obtained phase difference φ is higher. Therefore, in S254, the camera CPU 104 (first determination means) has a smaller result of a minimum correlation value between the A image and the B image (hereinafter also simply referred to as a minimum correlation value), that is, the A image and the B image. A result with a higher degree of coincidence of images is determined to be more reliable. Specifically, when the minimum value of the correlation value acquired from the image sensor 101 is less than or equal to the minimum value of the correlation value acquired from the image sensor 102, the camera CPU 104 has reliability of the signal acquired from the image sensor 101. It is determined that the reliability of the signal acquired from the image sensor 102 is higher. Further, when the minimum value of the correlation value acquired from the image sensor 101 is larger than the minimum value of the correlation value acquired from the image sensor 102, the camera CPU 104 determines that the reliability of the signal acquired from the image sensor 102 is the image sensor 101. It is determined that the reliability of the signal acquired from the above is higher. The camera CPU 104 selects a focus detection result based on a more reliable signal from the signal acquired from the image sensor 101 or the signal acquired from the image sensor 102, and the process proceeds to S255. Note that the focus detection results obtained from the two image sensors may be weighted so that more reliable focus detection results are used, and the focus detection results of both the two image sensors may be used.

  In S255, the camera CPU 104 converts the defocus amount, which is the focus detection result selected in S254, into a focus lens drive amount, and returns to the main flow in S256.

[Focus detection 4]
FIG. 15B is a flow of “focus detection 4” executed by the camera CPU 104 during moving image shooting in the image pickup apparatus according to the second embodiment.

  When the process shifts from S134 in the main flow shown in FIG. 12 to S271, the camera CPU 104 performs focus detection by the image sensor 102 that is driven for moving image shooting. Since this step is the same as S253 in FIG. In S271, the camera CPU 104 calculates the phase difference φ2 and the defocus amount DEF2 due to the image sensor 102. In S272, the still image pickup device 102 in the non-driven state of the camera CPU 104 is driven. In S273, the same control as S251 in FIG. In S273, the camera CPU 104 calculates the phase difference φ1 and the defocus amount DEF1 by the image sensor 101, and the process proceeds to S274.

  In S274, the camera CPU 104 (first determination unit) performs the same determination as in S254 of FIG. That is, the camera CPU 104 compares the reliability of the focus detection result obtained by the second image sensor obtained in S271 with the reliability of the focus detection result obtained by the image sensor 101 obtained in S273. In S274, when the camera CPU 104 (first determination unit) selects a more reliable focus detection result, the process proceeds to S275. In S275, the defocus amount selected by the camera CPU 104 in S247 is converted into a focus lens drive amount, and the process returns to the main flow in S276.

[Effect of Example 2]
In the second embodiment described above, the focus detection result of the image sensor 101 and the focus detection result of the image sensor 102 are compared, and the camera CPU 104 (first determination unit) adopts a more reliable result. Instead of selecting one result alternatively as illustrated in the present embodiment, the camera CPU 104 weights both focus detection results based on the reliability of the focus detection results, A new defocus amount may be calculated.

  According to the second embodiment, even when the information stored in the imaging apparatus, for example, the information of the lens pupil distance PL and the sensor pupil distance PS, or the shading correction information is insufficient, the camera CPU 104 has more accuracy. A high focus detection result can be selected.

[Example 3]
In Example 1 and Example 2, the image sensor 101 and the image sensor 102 differ only in the sensor pupil distance PS, and the size and the number of pixels of the pixel portion are the same in both image sensors. In Example 3 described below, the image sensor 101 has a large number of pixels in order to obtain high-pixel still image shooting, and the image sensor 302 (the first image element 302) has a relatively small number of pixels in order to increase the frame rate compared to the image sensor 101. The present invention is applied to an image pickup apparatus having two image pickup elements).

[Configuration of image sensor]
FIG. 16 is a diagram illustrating the configuration of the image sensor 101 (first image sensor) and the image sensor 302 (second image sensor) in the third embodiment. 2A is a plan view in the vicinity of the center of the image sensor 101, and FIG. 2B is a cross-sectional view of one pixel portion. The image sensor 101 according to the third embodiment is the same as the first image sensor illustrated in FIGS. 2A and 2B in the first embodiment. That is, the plurality of pixel portions included in the imaging element 101 are square pixel portions each having a size of 4 μm in the horizontal direction (x) and the vertical direction (y) on the imaging surface, and the structure of these pixel portions is as follows. Virtually all are the same. These pixel portions are 6000 pixels in the horizontal direction and 4000 pixels in the vertical direction, and have an effective pixel number of 24 million pixels. The size of the imaging region is 24 mm in the horizontal direction and 16 mm in the vertical direction. In each pixel portion, RGB color filters are arranged in a mosaic pattern.

  FIGS. 16C and 16D are a plan view and a cross-sectional view of the image sensor 302. The image pickup element 302 is an image pickup element for mainly taking a moving image. Each pixel portion has a size of 6 μm in both the horizontal direction (x) and the vertical direction (y), and these pixel portions are arranged with 3840 pixels in the horizontal direction and 2160 pixels in the vertical direction, and the number of effective pixels is 8.30 million. This is an image sensor for so-called 4K moving image of pixels. The size of the imaging region can be obtained by the same calculation as that of the first imaging device, and is 23.04 mm in the horizontal direction and 12.96 mm in the vertical direction. Similar to the image sensor 101, RGB color filters are arranged in a mosaic pattern in each pixel portion.

  Further, as can be seen from a comparison between FIGS. 4B and 4D, the image sensor 101 and the image sensor 302 have the same cross-sectional structure, but the dimensions of the constituent members are different. As a result, the size and baseline length of the pair of focus detection pupils in the image sensor 101 and the image sensor 302 are also different.

[Sensor pupil distance]
FIG. 17 is a diagram for explaining the relationship between the exit pupil of the photographing lens 500 and the focus detection pupil in the two image sensors, and corresponds to FIG. 4 of the first embodiment. The photographic lens 500 will be described as being the same as in the first embodiment. FIG. 17A is an explanatory diagram of the image sensor 101. Since the image sensor 101 in the third embodiment has the same configuration and the same dimensions as the first image sensor in the first embodiment, detailed description thereof is omitted. Therefore, Expressions 1 to 6 can be applied to various characteristics related to the first image sensor 101 as they are.

  FIG. 17B is an explanatory diagram of the image sensor 302. The pixel unit 3021 is a pixel unit disposed at the center of the imaging surface, that is, the image height x = 0, and the pixel unit 3022 is a pixel unit disposed near the end of the imaging surface, for example, the image height x = 10 mm. A principal point 3021p of the pixel portion 3021 is a reference position for considering the light refraction action of the microlens 3021c included in the pixel portion, and a plane perpendicular to the optical axis passing through the principal point 3021p is a scheduled image formation of the image sensor 302. This is a so-called imaging surface IP3.

  In the pixel portion 3021 at the center of the imaging surface, the height h3 between the principal point 3021p of the microlens 3021c and the upper surfaces of the pair of photoelectric conversion portions is the height of the pixel portion. The shape (optical power) of the microlens 3021c is set so that the focal point of the microlens 3021c substantially coincides with the upper surfaces of the photoelectric conversion unit 3021a and the photoelectric conversion unit 3021b. Therefore, a pair of back-projected images AP3a and AP3b corresponding to the pair of photoelectric conversion units 3021a and 3021b are formed on a sensor pupil plane SPL3 to be described later, and this becomes a focus detection pupil that defines a light beam at the time of focus detection.

  In the pixel portion 3021, the optical axis of the microlens 3021 c and the center of the pixel portion (the boundary portion center between the pair of photoelectric conversion portions) coincide with each other. However, in the pixel portion 3022 located at the end of the imaging surface IP3, the optical axis of the micro lens 3022c does not coincide with the center of the pixel portion, and is decentered by a predetermined amount dx3 closer to the optical axis of the photographing lens. Therefore, the principal ray S32c of the pixel portion 3022 (a line connecting the boundary between the pair of photoelectric conversion portions and the principal point of the microlens) is inclined by a predetermined angle ω3 with respect to the optical axis, and is only a predetermined distance from the imaging surface IP3. Crosses the optical axis at a separated point CS3. A virtual plane passing through the intersection CS3 and orthogonal to the optical axis is the sensor pupil plane SPL3, and a distance PS3 between the sensor pupil plane SPL3 and the imaging plane IP1 is a sensor pupil distance PS3. On the sensor pupil plane, the focus detection pupils of all the pixel portions substantially coincide.

The base line length of the image sensor 101 can be calculated by the formulas 1 to 3 as described above, but the base line length GP3 and the base line angle α31 of the image sensor 302 can be calculated in the same manner, and are represented by the following formulas.
GP3 / PS3 = GS3 / h3 (Formula 18)
GP3 = GS3 × (PS3 / h3) (Formula 19)
α31 = GS3 / h3 = GP3 / PS3 (Formula 20)

  In addition, the sensor pupil distance PS of the image sensor 101 can be calculated by Expressions 4 to 6, but the sensor pupil distance PS3 of the image sensor 302 can also be calculated in the same direction, as shown in the following expression.

X3 / (PS3 + h3) = dx3 / h3 (Formula 21)
PS3 = X3 × (h3 / dx3) (Formula 22)
dx3 = h3 × (X3 / PS3) (Formula 23)

  Here, X3 is the x coordinate of the pixel portion 3022, dx3 is the amount of eccentricity of the microlens 3022c, and PS3 is the sensor pupil distance PS.

  The pixel structure of the pixel portion 3022 and the dimensions of each portion are the same as those of the pixel portion 3021 except for the amount of eccentricity of the microlens. Therefore, a pair of backprojected images corresponding to the pair of photoelectric conversion units 3022a and 3022b is formed on the sensor pupil plane SPL3. The backprojected images are backprojected images AP3a and AP3b of the photoelectric conversion unit of the pixel unit 3021. It becomes substantially the same. That is, by setting the amount of eccentricity of the microlens of the pixel unit so that Expression 23 holds in an arbitrary pixel unit, the sensor pupil distance of all the pixel units becomes the sensor pupil distance PS3. That is, on the sensor pupil plane, the focus detection pupils of all the pixel portions are shared by AP3a and AP3b.

As described above, the image pickup device 101 and the image pickup device 302 included in the image pickup apparatus according to the third embodiment have substantially the same pixel portion structure, but the pixel portion has different planar dimensions and height direction dimensions, and the focus detection pupil. The base line length and sensor pupil distance PS are also different. However, also in Example 3, the magnitude relationship between the principal ray angle ω at the edge of the imaging surface and the sensor pupil distance PS is
ω1> ω3 (Formula 24)
PS1 <PS3 (Formula 25)
The relationship is established. That is, the eccentric amount of the microlens is set so that the sensor pupil distance PS differs between the image sensor 101 and the image sensor 302.

[Shooting process]
As the shooting process in the third embodiment, the shooting flow and the focus detection flow described in the first or second embodiment can be applied.

[Effect of Example 3]
In the first and second embodiments, the image sensor 101 and the image sensor 102 have the same number of pixels, but in this embodiment, the number of pixels of the image sensor 302 is relatively larger than the number of pixels of the image sensor 101. Few. As a result, the image sensor 302 can shoot a moving image with a higher frame rate than the image sensor 101.

  In the present embodiment, the same effects as in the first and second embodiments can be obtained by applying the photographing flow and the focus detection flow described in the first and second embodiments.

[Example 4]
In the first embodiment, the second embodiment, and the third embodiment, both of the two image sensors can output a signal for detecting a phase difference. On the other hand, in the fourth embodiment described below, an image pickup apparatus in which only one of the two image pickup elements 302 (second image pickup element) can output a signal for detecting an imaging surface phase difference. It is the Example which applied this invention to. The other image sensor 401 of the two image sensors corresponds to the image sensor 101 in the first embodiment, and the image pickup apparatus of the present embodiment replaces the AFE 114 in the first embodiment with an AFE 116 (third Image processing unit).

[Configuration of image sensor]
FIG. 18 is a diagram illustrating the configuration of the image sensor 401 (third image sensor) and the image sensor 302 (second image sensor) in the fourth embodiment. 2A is a plan view in the vicinity of the center of the image sensor 101, and FIG. 2B is a cross-sectional view of one pixel portion. The image sensor 401 in the fourth embodiment has the same size and the number of pixels as the image sensor 101 used in the first, second, and third embodiments, but the photoelectric conversion unit included in the pixel section is each pixel. No signal is acquired for phase difference detection.

  In FIG. 18A, the plurality of pixel portions included in the imaging element 401 are square pixel portions each having a size of 4 μm in the horizontal direction (x) and the vertical direction (y) on the imaging surface. The structure of the pixel portion is substantially the same. These pixel portions are 6000 pixel portions in the horizontal direction and 4000 pixels in the vertical direction, and have an effective pixel number of 24 million pixels. The size of the imaging region is 24 mm in the horizontal direction and 16 mm in the vertical direction. In each pixel portion, RGB color filters are arranged in a mosaic pattern. Each pixel unit includes a single photoelectric conversion unit 401a as shown in FIG.

[Sensor pupil distance]
FIG. 19 is a diagram for explaining the relationship between the exit pupil of the photographing lens 500 and the focus detection pupils in the two image sensors, and corresponds to FIG. 4 of the first embodiment and FIG. 17 of the third embodiment. The photographic lens 500 will be described as being the same as in the first embodiment. 19A is an explanatory diagram of the image sensor 401, and FIG. 19B is an explanatory diagram of the image sensor 302. The image sensor 302 in the fourth embodiment has the same configuration and the same dimensions as the image sensor 302 in the third embodiment. Therefore, detailed description is omitted. Hereinafter, the image sensor 401 will be described.

  In FIG. 19A, the pixel unit 4011 is arranged at the center of the imaging surface, that is, the pixel unit arranged at the image height x = 0, and the pixel unit 3022 is arranged near the end of the imaging surface, for example, the image height x = 10 mm. This is a pixel portion. The principal point 4011p of the pixel portion 4011 is a reference position for considering the light refraction action of the microlens 4011c included in the pixel portion, and a plane perpendicular to the optical axis passing through the principal point 4011p is the expected image formation of the image sensor 401. This is the surface, that is, the imaging surface IP4.

  In the pixel portion 4011 at the center of the imaging surface, the height h4 between the principal point 4011p of the microlens 4011c and the upper surface of the single photoelectric conversion portion is the height of the pixel portion. The shape (optical power) of the microlens 4011c is set so that the focal point of the microlens 4011c substantially coincides with the upper surface of the photoelectric conversion unit 4011a. Therefore, a single backprojection image AP4a corresponding to a single photoelectric conversion unit 4011a is formed on a sensor pupil plane SPL4 described later, and this becomes an imaging pupil that defines a light flux at the time of imaging.

  In the pixel portion 4011, the optical axis of the micro lens 4011c and the center of the pixel portion (sensitivity centroid of a single photoelectric conversion portion) coincide with each other. However, in the pixel portion 4012 located at the end of the imaging surface IP4, the optical axis of the micro lens 4012c and the center of the pixel portion do not coincide with each other, and are decentered by a predetermined amount dx4 near the optical axis of the photographing lens. Therefore, the principal ray S42c of the pixel portion 4012 (a line connecting the sensitivity center of gravity of the single photoelectric conversion portion and the principal point of the microlens) is inclined by a predetermined angle ω4 with respect to the optical axis, and the imaging surface IP4. Crosses the optical axis at a point CS4 separated by a predetermined distance from the optical axis. A virtual plane passing through the intersection CS4 and orthogonal to the optical axis is the sensor pupil plane SPL4, and a distance PS4 between the sensor pupil plane SPL4 and the imaging plane IP4 is the sensor pupil distance PS4. On the sensor pupil plane, the imaging pupils of all the pixel portions substantially coincide.

  Since the image sensor 401 has a single photoelectric conversion unit, and the center of the pixel unit and the sensitivity center of the photoelectric conversion unit coincide with each other, it does not have a pupil division function for phase difference detection. However, the sensor pupil distance can be defined by a method similar to that for an image sensor having a pupil division function. In other words, the sensor pupil distance PS of the image sensor 101 of the first embodiment can be calculated by Expressions 4 to 6. However, the sensor pupil distance PS4 of the image sensor 401 of the fourth embodiment can be calculated in the same direction as in the following expression. become.

X4 / (PS4 + h4) = dx4 / h4 (Formula 26)
PS4 = X4 × (h4 / dx4) (Formula 27)
dx4 = h4 × (X4 / PS4) (Formula 28)

  Here, X4 is the x coordinate of the pixel portion 4012, dx4 is the amount of eccentricity of the microlens 4012c, and PS4 is the sensor pupil distance PS4.

  The pixel structure of the pixel portion 4012 and the dimensions of each portion are the same as those of the pixel portion 4011 except for the amount of eccentricity of the microlens. Accordingly, a single backprojection image corresponding to the single photoelectric conversion unit 4012a is formed on the sensor pupil plane SPL4, and this backprojection image is substantially the same as the backprojection image AP4a of the photoelectric conversion unit of the pixel unit 4011. Are the same. That is, by setting the amount of eccentricity of the microlens of the pixel unit so that Equation 28 holds in any pixel unit, the sensor pupil distance of all the pixel units becomes the sensor pupil distance PS4, and on the sensor pupil plane The imaging pupils of all the pixel portions are shared by the AP 4a.

As described above, the image sensor 401 included in the image pickup apparatus according to the fourth embodiment does not have a pupil division function, the image sensor 302 has a pupil division function, and the sensor pupil distance PS of both image sensors is different. Also in Example 4, the magnitude relationship between the principal ray angles ω4 and ω3 in the imaging element 401 and the imaging element 302 at the edge of the imaging surface is
ω4> ω3 (Formula 29)
The magnitude relationship between the sensor pupil distances PS4 and PS3 in the first and second image sensors is
PS4 <PS3 (Formula 30)
The relationship is established. That is, the amount of eccentricity of the microlens is set so that the image sensor 401 and the image sensor 302 have different sensor pupil distances PS.

[Shooting process]
The procedure of the photographing process in the fourth embodiment is different from the main flowchart of the first embodiment shown in FIG. 12 in the subroutine of focus detection 1 in S114 and the subroutine of focus detection 2 in S134, and the other steps are substantially the same. Are the same. Therefore, only a flow different from that in the first embodiment will be described.

  At the time of still image shooting by the image sensor 401 in the fourth embodiment, the camera CPU 104 goes from S101 to S103 and S111 in FIG. 12, and a focus detection 5 subroutine suitable for still image shooting is executed at S114. FIG. 20A is a subroutine flow of focus detection 5 in the fourth embodiment, and shows a series of control contents in steps after S414.

  When the process proceeds from S414 to S451, the camera CPU 104 drives the moving image pickup element 302 in the non-driven state, and a pair of focus detection signals (A image in the focus detection region) from the signal acquired by the moving image pickup element 302. Signal and B image signal). In S452, the camera CPU 104 reads information corresponding to the F number and the lens pupil distance PL at the time of focus detection from the look-up table for shading correction, and performs shading correction on the A image signal and the B image signal. Next, the phase difference φ2 between the A image signal and the B image signal and the minimum value C2 of the interphase value are calculated by the interphase operation using the interphase arithmetic expression of Expression 15. Then, conversion information for calculating the defocus amount DEF is read from a predetermined lookup table, the defocus amount DEF is calculated using the above-described equation 17, and the process proceeds to S453.

  In S453, the reliability of the focus detection result obtained in S452 is calculated. In this embodiment, as an example, the reliability is obtained from the correlation value C (φ) described with reference to FIG. That is, the camera CPU 104 (second determination unit) has a more reliable focus detection result as the minimum value of the correlation value between the A image and the B image is smaller, or the inclination of the correlation value curve near the minimum value is larger. Judge as high. Specifically, in this embodiment, the minimum values of the A image and the B image are compared with a second predetermined value. Alternatively, the slope of the correlation value curve near the minimum value is compared with a third predetermined value. However, the determination index is not limited to this, and other indexes may be used. In S454, the reliability calculated in S453 is determined. When it is determined that the reliability is high (when the minimum value of the correlation value is equal to or smaller than the second predetermined value, or when the slope of the correlation value curve near the minimum value is larger than the third predetermined value), the process proceeds to S455. Transition. In S455, the defocus amount calculated in S452 is converted into a focus lens drive amount, and the process returns to the main flow in S458.

  In S454, the camera CPU 104 (second determination unit) has low reliability of the focus detection result (the minimum value of the correlation value is larger than the second predetermined value or the slope of the correlation value curve near the minimum value is the third value). If it is determined that the value is equal to or smaller than the predetermined value, the process proceeds to S456. In S456, the camera CPU 104 obtains information for performing contrast AF. That is, the camera CPU 104 detects a high frequency component in the image pickup signal obtained by adding the A image and the B image in the focus detection area. The camera CPU 104 (second focus detection unit) calculates an evaluation value for contrast AF based on the detection result of the high frequency component. Then, the camera CPU 104 transmits a focus lens scan command for contrast AF to the lens CPU 507 in S457 based on the calculated evaluation value, shifts to S458, and returns to the main routine.

  Note that the results based on the signals acquired from the image sensor 401 and the image sensor 302 may be weighted and used in accordance with the reliability determination result in S454. In this case, when the camera CPU 104 (second determination unit) determines that the reliability of the focus detection result is high, the result based on the signal acquired from the image sensor 302 is compared with the result based on the signal acquired from the image sensor 401. Adopt a lot (weight a lot). When the camera CPU 104 (second determination unit) determines that the reliability of the focus detection result is low, the result based on the signal acquired from the image sensor 401 is more than the result based on the signal acquired from the image sensor 302. Adopt a lot (weight a lot).

  At the time of moving image shooting by the image sensor 302 in the fourth embodiment, the camera CPU 104 executes a focus detection 6 subroutine suitable for moving image shooting at S134 via S101 to S103 and S131 in FIG. FIG. 20B is a subroutine flow of focus detection 6 in the fourth embodiment, and shows a series of control contents in steps after S434.

  After shifting from S434 to S471, the camera CPU 104 performs focus detection by the image sensor 302 that is driven for moving image shooting. The control content of S302 is the same as that of S452 in FIG. That is, the camera CPU 104 performs phase difference type focus detection with the image sensor 302 and calculates the defocus amount. In S472, the camera CPU 104 converts the defocus amount calculated in S471 into a focus lens drive amount, and returns to the main flow in S473.

[Effect of Example 4]
In the fourth embodiment described above, the sensor pupil distance PS suitable for the intended use of each image sensor is set as the sensor pupil distance PS of the image sensor 401 and the sensor pupil distance PS of the image sensor 302, which are different from each other. Value. Only the image sensor 302 mainly used for moving image shooting has a pupil division function for phase difference detection. Therefore, when adjusting the focus during still image shooting, first, phase difference focus detection is performed using the imaging device 302 for moving image shooting. If the focus detection result is highly reliable, the focus lens is driven using the phase difference detection result. On the other hand, if the reliability of the phase difference focus detection result is low, contrast AF (contrast detection focus adjustment) is performed. The reason for this is that it is not necessary to consider the influence of the focus lens operation when taking a still image, but focusing accuracy is the most important, so if the reliability of the phase difference focus detection result is low, it is more appropriate. This is because it is preferable to execute contrast AF with high focus accuracy. In contrast, since the contrast AF uses an imaging signal obtained by adding the A image signal and the B image signal, the decrease in focus detection accuracy due to the difference between the lens pupil distance PL and the sensor pupil distance PS is smaller than that of the phase difference focus detection. There is also a merit that there is.

  On the other hand, the reliability determination of the phase difference focus detection result is not performed during moving image shooting, and phase difference focus adjustment is always performed. The reason for this is that since AF at the time of moving image shooting is important not only for accuracy but also for operation quality, it is preferable to perform phase difference AF with good operation quality even if the focusing accuracy is expected to decrease somewhat.

  That is, according to the present embodiment, as in the first embodiment, the second embodiment, and the third embodiment, the sensor pupil distance PS is made different by comparing the sensor pupil distance between the two imaging elements. Thus, it is possible to cope with a change in the lens pupil distance PL in a wider range. Further, according to the present embodiment, by using two AF methods (phase difference AF and contrast AF) having different characteristics according to the situation, the image quality of the recorded image is ensured and more accurate focus detection is performed. be able to.

  Although the description has been made assuming that the pixel unit of the image sensor 402 of the present embodiment includes one photoelectric converter, each pixel unit has a plurality of photoelectric converters like the image sensor 302. Also good. Even in this case, it is possible to obtain the same effects as those described in the present embodiment.

[Modification]
In each of the imaging elements of the first embodiment, the second embodiment, the third embodiment, and the fourth embodiment described above, each pixel unit provided for phase difference detection includes a pair of photoelectric conversion units juxtaposed in the x direction. In this embodiment, the exit pupil of the photographing lens is divided into pupils in the x direction, and the phase difference in the x direction of the subject image is detected. On the other hand, each pixel unit of the image sensor is configured by m × n photoelectric conversion units in the x direction and the y direction along the imaging surface, and the x direction or y according to the directionality of the bright and dark pattern of the subject image. It is also possible to adopt a configuration in which a phase difference is detected in a desired direction among the directions or in both directions. In addition to the focus detection, the same effect can be achieved when applied to an imaging apparatus capable of acquiring light field information.

  In addition, in each of the image sensors of Examples 1 to 4, the image sensor having the pupil division function has a configuration in which all the pixel units have a plurality of photoelectric conversion units, and all the pixel units can detect the phase difference. It was. On the other hand, an imaging element in which a part of the imaging pixel unit is replaced with a focus detection pixel unit may be used. In this case, all the pixel units include a single photoelectric conversion unit, the imaging pixel unit receives a light beam that passes through the entire exit pupil of the photographing lens, and the focus detection pixel unit has an optical axis in the exit pupil region. On the other hand, the light beam in a part of the region decentered is received. The focus detection pixel unit includes, for example, a light-shielding unit that partially shields the photoelectric conversion unit, so that it can receive a light beam in a part of the exit pupil region that is decentered with respect to the optical axis. . The focus detection pixel units are arranged in a predetermined arrangement pattern between the imaging pixel units. Alternatively, a configuration in which half of all the pixel portions are for acquiring an A image signal and the other half is a pixel portion for acquiring a B image signal, and these pixel portions are alternately arranged.

  As described above, when the difference between the sensor pupil distance and the lens pupil distance (exit pupil distance) increases, the pixel portion at a position where the image height of the image sensor is high is compared with the pixel portion where the image height is low. Thus, the level difference between the A image signal and the B image signal becomes large. Therefore, in other words, in the pixel portion at the position where the image height is low, the A image signal and B due to the difference between the sensor pupil distance and the lens pupil distance (exit pupil distance) are compared with the pixel portion where the image height is high. The level difference of the image signal is small. From this, depending on the image height at which the pixel unit for acquiring the signal used for phase difference detection is located, it may be different which of the two image sensors is used for focus detection.

  FIG. 21 is a diagram for explaining a modification. In FIG. 21, Area 0 is the imaging range of the imaging element 101 or the imaging element 102, Area 1 is the periphery of the imaging area, and Area 2 is the center of the imaging area. When the phase difference is detected by using the signal of the pixel portion in the area where the image height is higher than the predetermined image height, that is, Area 1, as described in the first to fourth embodiments, focus detection is performed. An image sensor that acquires a signal to be used is selected. On the other hand, in the case where focus detection is performed using a signal of the pixel portion in the position where the image height is lower than the predetermined image height, that is, Area 2, in order to select an image sensor for acquiring a signal for focus detection There may be a case where a separate reference is provided, and an image sensor having a larger deviation is selected according to the reference. As an example, when a recorded image signal is acquired from an image sensor in which the difference between the sensor pupil distance and the lens pupil distance is larger than that of the other image sensor, the focus is obtained by using the signal of the pixel portion at a position lower than a predetermined image height. In the case of detection, focus detection may be performed using signals from the same image sensor. In this case, the power consumption is reduced as compared with the case of driving both image sensors. The area of Area 2 may be changed according to the F number of the photographing lens.

  In these modifications, the sensor lens distance PS is set between the two imaging elements by decentering the microlens included in each pixel unit as described in the first, second, third, and fourth embodiments. It is possible to obtain the same effect as in each embodiment.

101, 401 First imaging element 102, 302 Second imaging element 103 Beam splitter 500 Shooting lens 501 First lens group 502 Second lens group 503 Third lens group 1011, 1012, 1021, 1022, 3021, 3022, 4011 Pixel unit 1011a, 1011b, 1012a, 1012b, 1021a, 1021b, 1022a, 1022b, 3021a, 3021b, 3022a, 3022b, 4011a4012a photoelectric conversion unit 1011c, 1012c, 1021c, 1022c, 3021c, 3022c, 4011c, 4012c micro lens EPL1 lens pupil Surface (exit pupil surface)
SPL1, SPL2, SPL3, SPL4 Sensor pupil plane IP1, IP2, IP3, IP4 Planned imaging plane (imaging plane)

Claims (35)

Light beam splitting means for splitting the light beam that has passed through the taking lens into a plurality of light beams;
A first imaging element that has a plurality of pixel units each having a plurality of photoelectric conversion units that respectively receive and photoelectrically convert light beams having parallax with respect to one microlens, and receive one of the divided light beams;
One microlens has a plurality of pixel units each having a plurality of photoelectric conversion units that receive and photoelectrically convert a light beam having parallax, and receive the other of the divided light beams, and the length of the sensor pupil distance A second image sensor that is longer than the sensor pupil distance of the first image sensor;
1st focus detection means for calculating a defocus amount using two signals having parallax acquired from at least one of the first image sensor and the second image sensor apparatus.   The sensor pupil distance is a distance from an imaging surface to a sensor pupil surface located at a point where a principal ray of a pixel unit and an optical axis of a light beam passing through the photographing lens intersect. The imaging device described.   The imaging apparatus according to claim 1, wherein the principal ray of the pixel unit is a line connecting the center of the boundary between the plurality of photoelectric conversion units and the principal point of the microlens.   The first focus detection unit includes an imaging element that acquires a signal for calculating a defocus amount according to a lens pupil distance that is a distance between an exit pupil plane that is a plane on which the exit pupil is located and an imaging plane. The imaging device according to claim 1, wherein the imaging device is switched.   The first focus detection unit calculates a defocus amount by using an image sensor having a smaller difference between the lens pupil distance and the sensor pupil distance among the first image sensor and the second image sensor. The imaging apparatus according to claim 4, wherein: When the lens pupil distance is equal to or less than a first predetermined distance, the first focus detection unit calculates a defocus amount using two signals having parallax acquired from the first image sensor. And
When the lens pupil distance is longer than the first predetermined distance, the first focus detection unit calculates the defocus amount using two signals having parallax acquired from the second image sensor. 6. The imaging device according to claim 4, wherein the imaging device calculates the imaging device.   The imaging apparatus according to claim 6, wherein the first predetermined distance is longer than a sensor pupil distance of the first imaging element and shorter than a sensor pupil distance of the second imaging element.   The imaging apparatus according to claim 4, further comprising an acquisition unit that acquires the lens pupil distance from the photographing lens.   9. The sensor pupil distance between the first image sensor and the second image sensor is a distance within a range in which the lens pupil distance can be changed, according to claim 4. The imaging device described.   Depending on which of the reliability of the two signals having parallax acquired from the first image sensor and the reliability of the two signals having parallax acquired from the second image sensor is higher The image pickup apparatus according to claim 1, wherein the first focus detection unit switches an image pickup element used for calculating a defocus amount. When the reliability of two signals having parallax acquired from the first image sensor is higher than the reliability of two signals having parallax acquired from the second image sensor, the first signal The focus detection means calculates a defocus amount using two signals having parallax acquired from the first image sensor,
When the reliability of two signals having parallax acquired from the second image sensor is higher than the reliability of two signals having parallax acquired from the first image sensor, the first focus The image pickup apparatus according to claim 10, wherein the detection unit calculates a defocus amount using two signals having parallax acquired from the second image pickup element. When the minimum value of the correlation value of two signals having parallax acquired from the first image sensor is equal to or less than the minimum value of the correlation value of two signals having parallax acquired from the second image sensor The reliability of the two signals having parallax acquired from the first image sensor is higher than the reliability of the two signals having parallax acquired from the second image sensor;
When the minimum value of the correlation value of two signals acquired from the first image sensor is larger than the minimum value of the correlation value of two signals having parallax acquired from the second image sensor, the first value The reliability of two signals having parallax acquired from two image sensors is higher than the reliability of two signals having parallax acquired from the first image sensor. 11. The imaging device according to 11.   The imaging apparatus according to claim 1, wherein the light beam splitting unit splits the light beam by transmitting and reflecting the light beam that has passed through the photographing lens.   The first image sensor receives a light beam transmitted through the light beam splitting unit, and the second image sensor receives a light beam reflected by the light beam splitting unit. Imaging device.   15. The size of the plurality of pixel portions included in the first image sensor is smaller than the size of the plurality of pixel portions included in the second image sensor. The imaging apparatus of Claim 1. The number of the plurality of pixel units included in the second image sensor is less than the number of the plurality of pixel units included in the first image sensor,
The image pickup apparatus according to claim 1, wherein the second image pickup element has a higher frame rate than the first image pickup element.   The first image sensor mainly acquires a signal for generating a still image as a recorded image signal, and the second image sensor acquires a signal for mainly generating a moving image as a recorded image signal. The imaging device according to any one of claims 14 to 16, wherein A first image processing unit for processing a signal acquired from the first image sensor;
A second image processing unit for processing a signal acquired from the second image sensor,
The signal acquired from each of the first image sensor and the second image sensor by the first image processor and the second image processor is processed in parallel. The imaging device according to any one of claims 1 to 17.   The first image sensor and the second image sensor are disposed at optically equivalent positions, the distance from the imaging surface of the first image sensor to the exit pupil plane, and the second image sensor The imaging apparatus according to claim 1, wherein a distance from the imaging surface to the exit pupil plane is equal. Light beam splitting means for splitting the light beam that has passed through the taking lens into a plurality of light beams;
A second imaging element that has a plurality of pixel units each having a plurality of photoelectric conversion units that respectively receive and photoelectrically convert light beams having parallax with respect to one microlens, and receive one of the divided light beams;
A plurality of pixel units each having a photoelectric conversion unit that receives and photoelectrically converts a light beam, and receives the other of the divided light beams;
First focus detection means for calculating a defocus amount using a signal acquired from at least one of the second image sensor and the third image sensor;
When the reliability of the signal acquired from the second image sensor is high, the first focus detection unit uses the signal acquired from the second image sensor more than the signal acquired from the third image sensor. Use many to detect focus,
When the reliability of the signal acquired from the second image sensor is low, the first focus detection unit has more signals acquired from the third image sensor than signals acquired from the second image sensor. An imaging apparatus characterized by using the focus detection. When the minimum value of the correlation value of two signals having parallax acquired from the second image sensor is equal to or less than a second predetermined value, the two signals having parallax acquired from the second image sensor Reliable,
Reliability of two signals having parallax acquired from the third image sensor when the minimum value of the correlation value of the two signals having parallax acquired from the second image sensor is larger than a second predetermined value The imaging device according to claim 20, wherein the imaging device is low. The first focus detection unit calculates a defocus amount using a signal acquired from one of the second image sensor and the third image sensor,
When the reliability of the signal acquired from the second image sensor is high, the first focus detection unit calculates the defocus amount using the signal acquired from the second image sensor, and the second The first focus detection means calculates a defocus amount using a signal acquired from the third image sensor when the reliability of the signal acquired from the image sensor is low. Item 20. The imaging device according to Item 20 or Item 21. The first focus detection means is based on both the result based on the signal acquired from the second image sensor and the result based on the signal acquired from the third image sensor based on the signal acquired from each image sensor. Weighted according to the reliability of the result and used to detect the defocus amount,
When the reliability of the signal acquired from the second image sensor is high, the first focus detection unit obtains a result based on the signal acquired from the second image sensor rather than the third image sensor. When the reliability of the signal acquired from the second image sensor is low, the weight is obtained from the third image sensor rather than the second image sensor. The imaging apparatus according to claim 20 or 21, wherein a result based on the signal is weighted more and used for calculation of a defocus amount.   21. The plurality of pixel units of the third image sensor each include a plurality of photoelectric conversion units that receive and photoelectrically convert a light beam having parallax with respect to one microlens. Item 24. The imaging device according to any one of Items 23. Second focus detection means for calculating an evaluation value for contrast AF;
The said 2nd focus detection means performs a focus detection based on the evaluation value for contrast AF, when performing a focus detection using the signal acquired from the said 3rd image pick-up element. The imaging device according to any one of claims 20 to 24.   The length of the sensor pupil distance, which is the distance from the imaging surface of the second imaging device to the sensor pupil plane located at the point where the principal ray of the pixel unit and the optical axis of the light beam passing through the photographing lens intersect, The imaging device according to any one of claims 20 to 25, wherein the imaging device is shorter than the third imaging device.   27. The imaging apparatus according to claim 20, wherein the light beam splitting unit splits the light beam by transmitting and reflecting the light beam that has passed through the photographing lens.   28. The second imaging device according to claim 27, wherein the second image sensor receives a light beam reflected by the light beam splitting unit, and the third image sensor receives a light beam transmitted through the light beam splitting unit. Imaging device.   29. The size of the plurality of pixel portions included in the third image sensor is smaller than the size of the plurality of pixel portions included in the second image sensor. The imaging apparatus of Claim 1. The number of the plurality of pixel units included in the second image sensor is less than the number of the plurality of pixel units included in the third image sensor,
30. The image pickup apparatus according to claim 20, wherein the second image pickup element has a higher frame rate than the first image pickup element.   The second image sensor mainly acquires a signal for generating a moving image as a recorded image signal, and the first image sensor mainly acquires a signal for generating a still image as a recorded image signal. The imaging device according to any one of claims 28 to 30, wherein A second image processing unit for processing a signal acquired from the second image sensor;
A first image processing unit for processing a signal acquired from the third image sensor,
The signal acquired from each of the second image sensor and the third image sensor by the first image processor and the second image processor is processed in parallel. 32. The imaging device according to any one of claims 31 to 31.   The second image sensor and the third image sensor are disposed at optically equivalent positions, the distance from the imaging surface of the second image sensor to the exit pupil plane, and the third imaging The imaging apparatus according to any one of claims 20 to 32, wherein the distances from the imaging surface of the element to the exit pupil plane are equal. Light beam splitting means for splitting the light beam that has passed through the taking lens into a plurality of light beams;
A first imaging element that has a plurality of pixel units each having a plurality of photoelectric conversion units that respectively receive and photoelectrically convert light beams having parallax with respect to one microlens, and receive one of the divided light beams;
One microlens has a plurality of pixel units each having a plurality of photoelectric conversion units that receive and photoelectrically convert a light beam having parallax, and receive the other of the divided light beams, and the length of the sensor pupil distance In the control method of the imaging apparatus, wherein the second imaging element is longer than the sensor pupil distance of the first imaging element,
An image pickup apparatus comprising: a first focus detection step for calculating a defocus amount using two signals having parallax acquired from at least one of the first image pickup element and the second image pickup element. Light beam splitting means for splitting the light beam that has passed through the taking lens into a plurality of light beams;
A second imaging element that has a plurality of pixel units each having a plurality of photoelectric conversion units that respectively receive and photoelectrically convert light beams having parallax with respect to one microlens, and receive one of the divided light beams;
In a method for controlling an imaging apparatus, comprising: a plurality of pixel units each having a photoelectric conversion unit that receives and photoelectrically converts a light beam; and a third image sensor that receives the other of the divided light beams.
A first focus detection step of calculating a defocus amount using a signal acquired from at least one of the second image sensor and the third image sensor;
When the reliability of the signal acquired from the second image sensor is high, in the first focus detection step, the signal acquired from the second image sensor is more than the signal acquired from the third image sensor. Use many to detect focus,
When the reliability of the signal acquired from the second image sensor is low, the signal acquired from the third image sensor is larger than the signal acquired from the second image sensor in the first focus detection step. A method for controlling an imaging apparatus, wherein focus detection is performed using the imaging apparatus.