JP2020085920A - Imaging device and control method of the same - Google Patents

Abstract

To provide an imaging device that can acquire a focus detection result high in credibility even when errors are included in signals from a photoelectric conversion unit due to an impact of cross-talk, cross-talk correction and the like.SOLUTION: An imaging device is provided that has an image pick-up element 14 with a plurality of photoelectric conversion parts receiving a light flux passing through a different pupil area. The imaging device is configured to: acquire a signal of a first spectral characteristic and a signal of a second spectral characteristic from each of a first photoelectric conversion part and a second photoelectric conversion part; weight to a first amount of correlation to be calculated based on the signal of the first spectral characteristic and a second amount of correlation to be calculated based on the signal of the second spectral characteristic, and calculate a third amount of correlation; and, upon calculating the third amount of correlation, control a size of the weighting to the first amount of correlation and second amount of correlation in accordance with an image height of a focus detection area.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image pickup device and a control method thereof.

Patent Document 1 discloses an image pickup apparatus that performs focus detection by a pupil division method using an image pickup element in which a microlens is formed in each of two-dimensionally arranged pixels. This imaging device has a configuration in which one microlens is shared by two photoelectric conversion units. As a result, the first photoelectric conversion unit of the two photoelectric conversion units sharing the microlens outputs a signal based on the light flux emitted from the first region in the exit pupil of the photographing lens. Further, the second photoelectric conversion unit outputs a signal based on the light flux emitted from the second region in the exit pupil of the photographing lens. The imaging device performs a correlation calculation between a signal sequence obtained from the plurality of first photoelectric conversion units and a signal sequence obtained from the plurality of second photoelectric conversion units to calculate a phase difference (deviation amount) of the signal sequence. Then, the defocus amount is calculated based on the phase difference.

Further, by adding the outputs of the first photoelectric conversion unit and the second photoelectric conversion unit that share the microlens, it is possible to obtain an output similar to that of a general pixel having one photoelectric conversion unit per microlens. it can. Therefore, from one pixel, the output of the first photoelectric conversion unit (A signal), the output of the second photoelectric conversion unit (B signal), and the combined output of the first photoelectric conversion unit and the second photoelectric conversion unit It is possible to obtain three types of output (A+B signal). The image pickup device disclosed in Patent Document 1 reads the A+B signal after reading the output (for example, the A signal) of one photoelectric conversion unit, and subtracts the A signal from the A+B signal without individually reading the B signal. Generate with. As a result, three types of signals can be acquired by reading out twice. Further, Patent Document 2 discloses an imaging device that performs focus detection after performing a process of removing an influence of crosstalk from a peripheral photoelectric conversion unit on an output of the photoelectric conversion unit. The process of removing the influence of crosstalk is called crosstalk correction.

JP, 2014-182360, A JP, 2009-122524, A JP, 2014-187067, A

The imaging device described in Patent Document 2 that performs focus detection by performing crosstalk correction on the output of the photoelectric conversion unit has the following problems. The amount of crosstalk generated varies depending on not only the output amount of pixels that cause crosstalk, but also the angle of incident light, the F value, the image height of the focus detection region, the area of the photoelectric conversion unit, the distance, and the like. Therefore, the image pickup device disclosed in Patent Document 2 cannot accurately perform crosstalk correction, and an error occurs in crosstalk correction, so that it is difficult to obtain a reliable focus detection result. It is an object of the present invention to provide an imaging device capable of obtaining a highly reliable focus detection result even when an error is included in a signal from a photoelectric conversion unit due to the influence of crosstalk or crosstalk correction.

An image pickup apparatus according to an embodiment of the present invention includes an image pickup device having a plurality of photoelectric conversion units that receive light fluxes passing through mutually different pupil regions of an image pickup optical system, and a light flux that passes through a first pupil region. An acquisition unit that acquires the signal of the first spectral characteristic and the signal of the second spectral characteristic from each of the first photoelectric conversion unit and the second photoelectric conversion unit that receives the light flux passing through the second pupil region. , A third correlation amount calculated based on the signal of the first spectral characteristic and a second correlation amount calculated based on the signal of the second spectral characteristic are weighted, And a control unit that calculates the correlation amount of. The control means controls the magnitude of the weight for the first correlation amount and the second correlation amount according to the image height of the focus detection area.

According to the imaging device of the present invention, a highly reliable focus detection result can be obtained even when an error is included in the signal from the photoelectric conversion unit due to the influence of crosstalk or crosstalk correction.

It is a figure which shows the structural example of an imaging device. It is a figure which shows the structural example of an image sensor. It is a figure which shows the detail of a structure of an image sensor. It is a figure which shows the timing chart regarding the read-out operation of an image sensor. It is a figure explaining the projection image of the photoelectric conversion part on an exit pupil surface. It is a figure explaining the characteristic of the incident angle photosensitivity distribution for every spectroscopy of a photoelectric conversion part. It is a figure explaining the relationship of a 1st pupil area, a 2nd pupil area, and a 3rd pupil area. It is a figure which shows an example of setting of a focus detection area. It is a figure which shows the example of the pixel arrange|positioned in a focus detection area. It is a line image intensity distribution regarding a signal obtained from a pixel in the focus detection area. It is a line image intensity distribution regarding a signal obtained from a pixel in the focus detection area. It is a figure explaining an example of crosstalk between pixels. It is a flow chart explaining an example of focus adjustment operation. 14 is a flowchart illustrating the processing of S504 and S504 of FIG. It is a figure explaining the subroutine of correlation amount calculation. It is a figure explaining the subroutine of correlation amount calculation. 7 is a flowchart illustrating a defocus amount calculation process.

FIG. 1 is a diagram showing a configuration example of the image pickup apparatus of the present embodiment.
In FIG. 1, a lens-interchangeable digital single-lens reflex camera will be described as an example of an imaging device including a focus detection device. INDUSTRIAL APPLICABILITY The present invention can be applied to any electronic device having an image sensor capable of generating a signal used for focus detection of a phase difference detection method. Electronic devices to which the present invention is applicable include, for example, cameras such as digital still cameras and digital video cameras, mobile phones having a camera function, computer devices, media players, robot devices, game devices, home appliances, and the like. It is not limited to the device.

The imaging device shown in FIG. 1 includes a camera 100 and a taking lens 300 that is attachable to and detachable from the camera. The light flux that has passed through the taking lens 300 passes through the lens mount 106, is reflected upward by the main mirror 130, and enters the optical finder 104. The optical viewfinder 104 allows a photographer to take a picture while observing an optical image of the subject. The optical finder 104 has a part of functions of the display unit 54, for example, functions related to focus display, camera shake warning display, aperture value display, and exposure correction display.

A part of the main mirror 130 is a semi-transmissive half mirror. A part of the light flux incident on the main mirror 130 passes through the half mirror portion, is reflected downward by the sub mirror 131, and is incident on the focus detection device 105. The focus detection device 105 is a phase difference detection type focus detection device having a secondary imaging optical system and a line sensor, and outputs a pair of image signals to the AF unit (autofocus unit) 42. The AF unit 42 performs a phase difference detection calculation on the pair of image signals to obtain the defocus amount and direction of the taking lens 300. The system control unit 50 controls a focus control unit 342 (described later) of the taking lens 300 based on the result of the phase difference detection calculation to control the drive of the focus lens.

When the still image shooting is performed after the focus adjustment processing of the taking lens 300 is completed, the electronic viewfinder display is performed, and the moving image shooting is performed, the system control unit 50 uses the quick return mechanism (not shown) to operate the main mirror 130 and the main mirror 130. The sub mirror 131 is retracted out of the optical path. As a result, the light flux that passes through the taking lens 300 and enters the camera 100 can enter the image sensor 14 via the shutter 12 for controlling the exposure amount. After the image capturing operation by the image sensor 14 is completed, the main mirror 130 and the sub mirror 131 return to the positions as illustrated.

The image sensor 14 is a CCD or CMOS image sensor. CCD is an abbreviation for Charge Coupled Device. CMOS is an abbreviation for Complementary Metal Oxide Semiconductor. The image sensor 14 has a configuration in which a plurality of pixels each having a photoelectric conversion unit (for example, a photodiode) are two-dimensionally arranged. The image sensor 14 outputs an electric signal corresponding to the subject optical image. The electric signal photoelectrically converted by the image sensor 14 is sent to the A/D converter 16, and the analog signal output is converted into a digital signal (image data). The A/D converter 16 may be incorporated in the image sensor 14.

In the image sensor 14, at least some of the pixels have a plurality of photoelectric conversion units (first photoelectric conversion unit and second photoelectric conversion unit). The first photoelectric conversion unit and the second photoelectric conversion unit receive light beams passing through different pupil regions of the image pickup optical system. A pixel having such a configuration can output a signal used for focus detection by the phase difference detection method. Therefore, even when the main mirror 130 and the sub mirror 131 are retracted out of the optical path by the quick return mechanism and light does not enter the focus detection device 105, the system control unit 50 uses the phase difference using the output of the image pickup device 14. It is possible to execute the focus detection processing of the detection method. That is, the image sensor 14 has a function of a focus detection sensor for phase difference AF in addition to a function of acquiring a captured image. Since signals obtained by a plurality of photoelectric conversion units can be combined for each pixel and used as an output of a normal image pickup pixel, contrast AF can be performed using the output of the image pickup device 14 (pickup image signal).

The timing generation circuit 18 supplies a clock signal and a control signal to the image sensor 14, the A/D converter 16, and the D/A converter 26. The timing generation circuit 18 is controlled by the memory control unit 22 and the system control unit 50. The system control unit 50 controls the timing generation circuit 18 to read the output of a part of the photoelectric conversion areas from the pixel having the plurality of photoelectric conversion areas, or to read the outputs of all the photoelectric conversion areas by addition. To the image sensor 14.

The image processing unit 20 applies predetermined processing such as pixel interpolation processing, white balance adjustment processing, and color conversion processing to the image data from the A/D converter 16 or the image data from the memory control unit 22. Further, the image processing unit 20 performs focus detection of the phase difference detection method from the output signal used for generating the focus detection signal in the image data (output signal of the image pickup device 14) from the A/D converter 16. Generate a pair of signal sequences to be used. The pair of signal trains is sent to the AF unit 42 via the system control unit 50. The AF unit 42 detects a shift amount (shift amount) between the signal sequences by performing a correlation calculation of the pair of signal sequences, and converts the shift amount into the defocus amount and the defocus direction of the taking lens 300. The AF unit 42 outputs the converted defocus amount and direction to the system control unit 50. The system controller 50 drives the focus lens through the focus controller 342 of the taking lens 300 to adjust the focusing distance of the taking lens 300.

Further, the image processing unit 20 can calculate the contrast evaluation value based on the signal (corresponding to the A+B signal described above) obtained from the image sensor 14 for generating normal image data. The system control unit 50 shoots with the image sensor 14 while changing the focus lens position through the focus control unit 342 of the shooting lens 300, and checks the change in the contrast evaluation value calculated by the image processing unit 20. Then, the system control unit 50 drives the focus lens to a position where the contrast evaluation value is maximum. As described above, the camera 100 according to the present embodiment can also perform focus detection by the contrast detection method. Therefore, even if the main mirror 130 and the sub mirror 131 are retracted out of the optical path such as during live view display or moving image shooting, the camera 100 uses the phase difference detection method and the contrast based on the signal obtained from the image sensor 14. Both focus detection methods are possible. In addition, the camera 100 can perform focus detection by the phase difference detection method by the focus detection device 105 in normal still image shooting in which the main mirror 130 and the sub mirror 131 are in the optical path. As described above, the camera 100 can perform focus detection in any state of still image shooting, live view display, and moving image shooting.

The memory control unit 22 controls the A/D converter 16, the timing generation circuit 18, the image processing unit 20, the image display memory 24, the D/A converter 26, the memory 30, and the compression/decompression unit 32. Then, the data of the A/D converter 16 is written in the image display memory 24 or the memory 30 via the image processing unit 20 and the memory control unit 22, or only via the memory control unit 22. The image data for display written in the image display memory 24 is displayed on the image display unit 28 having a liquid crystal monitor or the like via the D/A converter 26. An electronic viewfinder function (live view display) can be realized by sequentially displaying moving images captured by the image sensor 14 on the image display unit 28. The image display unit 28 can turn on/off the display according to an instruction from the system control unit 50. When the image display unit 28 turns off the display, the power consumption of the camera 100 can be significantly reduced.

The memory 30 is used for temporary storage of captured still images or moving images, and has a storage capacity sufficient to store a predetermined number of still images or moving images for a predetermined time. As a result, even in the case of continuous shooting or panoramic shooting, a large amount of images can be written in the memory 30 at high speed. The memory 30 can also be used as a work area for the system controller 50. The compression/decompression unit 32 has a function of compressing/decompressing image data by adaptive discrete cosine transform (ADCT) or the like, reads an image stored in the memory 30, performs compression processing or decompression processing, and completes the image data. Is written back to the memory 30.

The shutter control unit 36 controls the shutter 12 based on the photometric information from the photometric unit 46 in cooperation with the aperture control unit 344 that controls the aperture 312 of the taking lens 300. The interface unit 38 and the connector 122 electrically connect the camera 100 and the taking lens 300. The interface unit 38 and the connector 122 have a function of transmitting control signals, a status signal, a data signal, and the like between the camera 100 and the taking lens 300 and also supplying a current of various voltages. Further, the interface unit 38 and the connector 122 may transmit not only electric communication but also optical communication, voice communication and the like.

The photometric unit 46 performs automatic exposure control (AE) processing. The light flux that has passed through the taking lens 300 is incident on the photometric unit 46 via the lens mount 106, the main mirror 130, and a photometric lens (not shown), so that the brightness of the subject optical image can be measured. The photometric unit 46 can determine the exposure condition by using a program diagram or the like in which the subject brightness and the exposure condition are associated with each other. The photometric unit 46 also has a dimming processing function by cooperating with the flash 48. The system control unit 50 performs AE control on the shutter control unit 36 and the aperture control unit 344 in the taking lens 300 based on the calculation result of the image data of the image sensor 14 calculated by the image processing unit 20. Is also possible. The flash 48 also has an AF auxiliary light projection function and a flash light control function.

The system control unit 50 has a programmable processor such as a CPU or MPU, and controls the overall operation of the imaging device by executing a program stored in advance. The non-volatile memory 52 stores constants, variables, programs, etc. for the operation of the system controller 50. The display unit 54 displays an operating state, a message, and the like using characters, images, sounds, and the like according to the execution of the program by the system control unit 50. The display unit 54 is, for example, a liquid crystal display device. One or a plurality of display units 54 are installed near the operation unit of the camera 100 so as to be easily visible, and are configured by, for example, a combination of LCDs and LEDs. Of the display contents of the display unit 54, what is displayed on the LCD or the like is information regarding the number of shots such as the number of recorded images and the number of remaining shots, information regarding shooting conditions such as shutter speed, aperture value, exposure compensation, flash, etc. There is. In addition, the battery level, date and time, etc. are also displayed. Further, the display unit 54 is provided with a part of its function in the optical finder 104.

The nonvolatile memory 56 is an electrically erasable/recordable memory. The non-volatile memory 56 is, for example, an EEPROM or the like. The operation units 60, 62, 64, 66, 68 and 70 are operation units for inputting various operation instructions of the system control unit 50. The operation unit is realized by one or a combination of a switch, a dial, a touch panel, pointing by visual axis detection, a voice recognition device, and the like.

The mode dial 60 can switch and set each function mode such as power off, automatic shooting mode, manual shooting mode, playback mode, and PC connection mode. The shutter switch (SW1) 62 is turned on when a shutter button (not shown) is half-pressed to instruct the start of operations such as AF processing, AE processing, AWB processing, and EF processing. The shutter switch (SW2) 64 is turned on when the shutter button is fully pressed, and gives an instruction to start the operation of a series of processing related to shooting. The series of processes relating to shooting is an exposure process, a development process, a recording process, and the like. In the exposure processing, the signal read from the image sensor 14 is written as image data in the memory 30 via the A/D converter 16 and the memory control unit 22. In the developing process, the developing process is performed by using the calculation in the image processing unit 20 and the memory control unit 22. In the recording process, image data is read from the memory 30, compressed by the compression/expansion unit 32, and written as image data in the recording medium 150 or 160.

The image display ON/OFF switch 66 is an operation unit for setting ON/OFF of the image display unit 28. By the ON/OFF setting function of the image display unit 28, power can be saved by shutting off current supply to the image display unit 28 having a liquid crystal monitor or the like when shooting is performed using the optical finder 104. it can. The quick review ON/OFF switch 68 is an operation unit for setting a quick review function that automatically reproduces captured image data immediately after capturing. The operation unit 70 has various buttons, a touch panel, and the like. Various buttons include a menu button, a flash setting button, a single shooting/continuous shooting/self-timer switching button, an exposure correction button, and the like.

The power supply control unit 80 has a battery detection circuit, a DC/DC converter, a switch circuit that switches blocks to be energized, and the like. The power supply control unit 80 detects whether or not a battery is installed, the type of battery, and the remaining battery level, and controls the DC/DC converter based on the detection result and the instruction from the system control unit 50 to obtain the required voltage. For various periods, including the recording medium. The connectors 82 and 84 connect the camera 100 to the power supply unit 86 having a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery or a NiMH battery, a lithium ion battery, or an AC adapter.

The interfaces 90 and 94 have a function of connecting to a recording medium such as a memory card or a hard disk. The connectors 92 and 96 physically connect to a recording medium such as a memory card or a hard disk. The recording medium attachment/detachment detection unit 98 detects whether or not a recording medium is attached to the connectors 92 and 96. In the present embodiment, the imaging device has two systems of an interface and a connector for attaching a recording medium, but the system is not limited to two systems. Further, the image pickup apparatus may be provided with a combination of interfaces and connectors of different standards. By connecting various communication cards such as a LAN card to the interface and the connector, image data and management information attached to the image data can be transferred to and from other peripheral devices such as a computer and a printer. ..

The communication unit 110 has various communication functions such as wired communication and wireless communication. The connector 112 connects the camera 100 to another device via the communication unit 110, and is an antenna in the case of wireless communication. The recording media 150 and 160 are memory cards, hard disks, and the like. The recording media 150 and 160 are provided with recording units 152 and 162 composed of semiconductor memories and magnetic disks, interfaces 154 and 164 with the camera 100, and connectors 156 and 166 for connecting with the camera 100.

Next, the taking lens 300 will be described. The taking lens 300 is mechanically and electrically coupled to the camera 100 by engaging the lens mount 306 with the lens mount 106 of the camera 100. The electrical coupling is realized by the connectors 122 and 322 provided on the lens mount 106 and the lens mount 306. The lens 311 includes a focus lens for adjusting the focusing distance of the taking lens 300. The focus control unit 342 adjusts the focus of the taking lens 300 by driving the focus lens along the optical axis. The operation of the focus controller 342 is controlled by the system controller 50 as an adjusting unit through the lens system controller 346. The diaphragm 312 adjusts the amount and angle of subject light incident on the camera 100.

The connector 322 and the interface 338 electrically connect the taking lens 300 to the connector 122 of the camera 100. The connector 322 has a function of transmitting control signals, status signals, data signals, and the like between the camera 100 and the taking lens 300, and also being supplied with currents of various voltages. The connector 322 may transmit not only electric communication but also optical communication, voice communication and the like.

The zoom controller 340 drives the variable power lens of the lens 311 and adjusts the focal length (angle of view) of the taking lens 300. The aperture control unit 344 controls the aperture 312 in cooperation with the shutter control unit 36 that controls the shutter 12 based on the photometric information from the photometric unit 46. The lens system control unit 346 has a programmable processor such as a CPU or MPU, and controls the overall operation of the taking lens 300 by executing a program stored in advance. The lens system control unit 346 has a memory function of storing constants, variables, programs, etc. for operating the photographing lens. The non-volatile memory 348 stores identification information such as a number unique to the taking lens, management information, functional information such as an open aperture value or a minimum aperture value, a focal length, and current and past set values. In the present embodiment, the nonvolatile memory 348 also stores lens frame information according to the state of the taking lens 300. The lens frame information is information on the radius of the frame opening that determines the light flux passing through the taking lens and information on the distance from the image sensor 14 to the frame opening. The diaphragm 312 is included in the frame that determines the light flux that passes through the taking lens 300. Further, an opening of a lens frame component that holds the lens also corresponds to a frame that determines the light flux that passes through the taking lens 300. Further, the frame that determines the light flux passing through the taking lens differs depending on the focus position and zoom position of the lens 311. Therefore, a plurality of pieces of lens frame information are prepared corresponding to the focus position and zoom position of the lens 311. Then, when the camera 100 performs focus detection using the focus detection means, optimum lens frame information corresponding to the focus position and zoom position of the lens 311 is selected and sent to the camera 100 via the connector 322.

FIG. 2 is a diagram showing a configuration example of the image sensor.
FIG. 2A is a circuit configuration example of a pixel having a configuration capable of outputting a signal used for focus detection by a phase difference detection method, out of a plurality of pixels included in the image sensor 14. In FIG. 2, in one pixel 200, as a plurality of photoelectric conversion regions or photoelectric conversion units sharing one microlens, a photodiode (PD) 201a that is a first photoelectric conversion unit and a second photoelectric conversion unit. The photodiode (PD) 201b is provided. However, the pixel 200 may be provided with more PDs (for example, four). The PDs 201a and 201b function not only as focus detection pixels but also as imaging pixels.

The transfer switches 202a and 202b, the reset switch 205, and the selection switch 206 have, for example, MOS transistors. In the following description, these switches are N-type MOS transistors, but may be P-type MOS transistors or other switching elements.

FIG. 2B shows horizontal n pixels and vertical m pixels among the plurality of pixels two-dimensionally arranged on the image sensor 14. All the pixels shown in FIG. 2B have the structure shown in FIG. A microlens 236 is provided in each pixel, and the PDs 201a and 201b share the same microlens. Hereinafter, the signal obtained by the PD 201a will be described as an A signal or a first signal, and the signal obtained by the PD 201b will be described as a B signal or a second signal. A signal sequence for focus detection generated from a plurality of A signals is described as an A image or a first image signal. A signal sequence for focus detection generated from a plurality of B signals is described as a B image or a second image signal. In addition, the A image and the B image forming a pair are described as a signal sequence pair or an image signal pair.

The pixels arranged two-dimensionally on the image sensor 14 have color filters corresponding to any of a plurality of colors having different spectral characteristics. In the present embodiment, the image sensor 14 has rows in which Gr pixels and R pixels are alternately arranged and rows in which B pixels and Gb pixels are alternately arranged. .. The Gr pixel and the Gb pixel are pixels having a green (G) color filter. The R pixel is a pixel having a red (R) color filter. The first photoelectric conversion unit and the second photoelectric conversion unit of the Gr pixel are described as A-Gr and B-Gr, respectively. That is, A-Gr is a Gr pixel of the A image, and B-Gr is a Gr pixel of the B image. Further, the first photoelectric conversion unit of the R pixel is described as A-R, and the second photoelectric conversion unit thereof is described as B-R. That is, A-R is the R pixel of the A image, and B-R is the R pixel of the B image.

Further, the first photoelectric conversion unit of the B pixel is described as AB, and the second photoelectric conversion unit of the B pixel is described as BB. That is, AB is the B pixel of the A image, and BB is the B pixel of the B image. Further, the first photoelectric conversion unit of the Gb pixel is described as A-Gb and the second photoelectric conversion unit is described as B-Gb. That is, A-Gb is a Gb pixel of the A image, and B-Gb is a Gb pixel of the B image. The same notation is also applied to the signals obtained from the respective photoelectric conversion units. Further, a group of the first photoelectric conversion units is described as a first photoelectric conversion unit group, and a group of the second photoelectric conversion units is described as a second photoelectric conversion unit group.

The transfer switch 202a is connected between the PD 201a and the floating diffusion (FD) 203. The transfer switch 202b is connected between the PD 201b and the FD 203. The transfer switches 202a and 202b are elements that transfer the charges generated in the PDs 201a and 201b to the common FD 203, respectively. The transfer switches 202a and 202b are controlled by control signals TX_A and TX_B, respectively.

The floating diffusion (FD) 203 temporarily holds the charges transferred from the PDs 201a and 201b, and also functions as a charge-voltage converter (capacitor) that converts the held charges into a voltage signal. The amplification unit 204 is a source follower MOS transistor. The gate of the amplification unit 204 is connected to the FD 203, and the drain of the amplification unit 204 is connected to the common power supply 208 that supplies the power supply potential VDD. The amplification unit 204 amplifies the voltage signal based on the charges held in the FD 203 and outputs it as an image signal. The reset switch 205 is connected between the FD 203 and the common power source 208. The reset switch 205 is controlled by the control signal RES and has a function of resetting the potential of the FD 203 to the power supply potential VDD. The selection switch 206 is connected between the source of the amplification unit 204 and the vertical output line 207. The selection switch 206 is controlled by the control signal SEL and outputs the image signal amplified by the amplification unit 204 to the vertical output line 207.

FIG. 3 is a diagram showing details of the configuration of the image sensor.
The image sensor 14 includes a pixel array 234, a vertical scanning circuit 209, a current source load 210, a reading circuit 235, common output lines 228 and 229, a horizontal scanning circuit 232, and a data output section 233. It is assumed that all the pixels included in the pixel array 234 have the circuit configuration shown in FIG. Note that some pixels included in the pixel array 234 may have a configuration in which one PD is provided for each microlens.

The pixel array 234 has a plurality of pixels 200 arranged in a matrix. In FIG. 3, a pixel array 234 of 4 rows and n columns is shown for simplification of description. However, the number of rows and the number of columns of the pixels 200 included in the pixel array 234 are arbitrary. Further, in the present embodiment, the image pickup device 14 is a single-plate color image pickup device and has a color filter of primary color Bayer array. Therefore, the pixel 200 is provided with a color filter corresponding to any of a plurality of colors (red (R), green (G), and blue (B)) having different spectral characteristics. There are no particular restrictions on the colors or arrangement of the color filters. In addition, some pixels included in the pixel array 234 are shielded from light and form an optical black (OB) region.

The vertical scanning circuit 209 supplies various control signals to the pixels 200 in each row via the drive signal line 208 provided for each row. Note that, in FIG. 3, the drive signal line 208 of each row is represented by one line for simplification of description, but in reality, a plurality of drive signal lines are present in each row. The pixels included in the pixel array 234 are connected to the common vertical output line 207 for each column. A current source load 210 is connected to each of the vertical output lines 207. The signal from each pixel 200 is input to the readout circuit 235 provided for each column through the vertical output line 207.

The horizontal scanning circuit 232 outputs control signals hsr(0) to hsr(n-1) signals, each of which corresponds to one read circuit 235. The control signal hsr() selects one of the n read circuits 235. The read circuit 235 selected by the control signal hsr() outputs a signal to the data output unit 233 through the common output lines 228 and 229.

Next, a specific circuit configuration example of the read circuit 235 will be described. Although FIG. 3 shows a circuit configuration example of one of the n read circuits 235, the other read circuits 235 also have the same configuration. The readout circuit 235 of this embodiment includes a ramp type AD converter.

The signal input to the readout circuit 235 through the vertical output line 207 is input to the inverting input terminal of the operational amplifier 213 via the clamp capacitor 211. The reference voltage Vref is supplied from the reference voltage source 212 to the non-inverting input terminal of the operational amplifier 213. Feedback capacitors 214 to 216 and switches 218 to 220 are connected between the inverting input terminal and the output terminal of the operational amplifier 213. A switch 217 is further connected between the inverting input terminal and the output terminal of the operational amplifier 213. The switch 217 is controlled by the control signal RES_C and has a function of short-circuiting both ends of the feedback capacitors 214 to 216. The switches 218 to 220 are controlled by the control signals GAIN0 to GAIN2 from the system controller 50.

The output signal of the operational amplifier 213 and the ramp signal 224 output from the ramp signal generator 230 are input to the comparator 221. Latch_N222 is a memory element for holding a noise level (N signal). Latch_S is a storage element for holding the A signal and the signal level (A+B signal) obtained by adding the A signal and the B signal. The output of the comparator 221 (value indicating the comparison result) and the output of the counter 231 (counter value) 225 are input to the Latch_N 222 and Latch_S 223, respectively. The operation (valid or invalid) of Latch_N 222 and Latch_S 223 is controlled by LATEN_N and LATEN_S, respectively. The noise level held by Latch_N 222 is output to the common output line 228 via the switch 226. The signal level held by Latch_S223 is output to the common output line 229 via the switch 227. The common output lines 228 and 229 are connected to the data output unit 233.

The switches 226 and 227 are controlled by the control signal hsr(h) signal from the horizontal scanning circuit 232. h indicates the column number of the read circuit 235 to which the control signal line is connected. The signal levels held in the Latch_N 222 and Latch_S 223 of each read circuit 235 are sequentially output to the common output lines 238 and 229, and output to the memory control unit 22 and the image processing unit 20 through the data output unit 233. The operation of sequentially outputting the signal levels held by the read circuits 235 to the outside is called horizontal transfer. The control signals (excluding hsr()) input to the read circuit and the control signals of the vertical scanning circuit 209, the horizontal scanning circuit 232, the ramp signal generator 230, and the counter 231 are the timing generation circuit 18 and the system control unit. Supplied from 50.

FIG. 4 is a diagram showing a timing chart regarding the read operation of the image sensor.
The read operation for the pixels of one row will be described with reference to FIG. Note that each switch is turned on when each control signal is H (High), and each switch is turned off when it is L (Low). At time t1, the vertical scanning circuit 209 sets the control signals TX_A and TX_B from L to H with the control signal RES set to H, and turns on the transfer switches 202a and 202b. As a result, the charges accumulated in the PDs 201a and 201b are transferred to the power supply 208 via the transfer switches 202a and 202b and the reset switch 205, and the PDs 201a and 201b are reset. Also, the FD 203 is reset in the same manner. At time t2, the vertical scanning circuit 209 sets the control signals TX_A and TX_B to L and turns off the transfer switches 202a and 202b, so that the PDs 201a and 201b start accumulating photocharges.

When the predetermined accumulation time has elapsed, at time t3, the vertical scanning circuit 209 sets the control signal SEL to H and turns on the selection switch 206. As a result, the source of the amplifier 204 is connected to the vertical output line 207. At time t4, the vertical scanning circuit 209 sets the control signal RES to L and turns off the reset switch 205. As a result, the reset of the FD 203 is released, and the reset signal level of the FD 203 is read to the vertical output line 207 via the amplification unit 204 and input to the read circuit 235.

At time t5, the timing generation circuit 18 sets the control signal RES_C to L. As a result, the switch 217 is turned on, and the operational amplifier 213 outputs a voltage based on the difference between the reset signal level read on the vertical output line 207 and the reference voltage Vref. In the image pickup device 14, the system control unit 50 sets any one of the control signals GAIN0 to GAIN2 to H based on the ISO sensitivity set by the operation unit 102 in advance. For example, it is assumed that the camera 100 can set any one of ISO sensitivity 100, 200, and 400. When the ISO sensitivity 100 is set, the control signal GAIN0 becomes H, and the GAIN1 and GAIN2 become L. Similarly, when the ISO sensitivity 200 is set, the control signal GAIN1 becomes H. When the ISO sensitivity 400 is set, the control signal GAIN2 becomes H. The type of setting sensitivity and the relationship between the setting sensitivity and the control signal are not limited to the above.

The operational amplifier 213 amplifies the input voltage with an inverting gain determined by the capacitance ratio of the clamp capacitor 211 and one of the feedback capacitors 214 to 216 corresponding to the switch corresponding to H of the control signals GAIN0 to GAIN2. Output. By this amplification, the random noise component generated in the circuits up to the operational amplifier 213 is also amplified. Therefore, the magnitude of random noise included in the amplified signal depends on the ISO sensitivity.

Next, at time t6, the ramp signal generator 230 starts outputting a ramp signal whose signal level increases linearly with the passage of time, and at the same time, the counter 231 starts counting up from the reset state. Further, the timing generation circuit 18 sets LATEN_N to H to enable Latch_N. The comparator 221 compares the output signal of the operational amplifier 213 with the ramp signal output by the ramp signal generator 230. When the ramp signal level exceeds the output signal level of the operational amplifier 213, the output of the comparator 221 changes from L to H (time t7). When the output of the comparator 221 changes from L to H while LATEN_N is H, the Latch_N 222 stores the counter value output by the counter 231 at that time. The counter value stored in Latch_N222 corresponds to a digital value (N signal data) representing the N signal level. Since LATEN_S is L, Latch_S223 is invalid and the count value is not stored. Then, at time t8, when the ramp signal level reaches a predetermined value, the ramp signal generator 230 stops the output of the ramp signal, and the timing generation circuit sets LATEN_N to L.

At time t9, the vertical scanning circuit 209 sets the control signal TX_A to H. As a result, the transfer switch 202a is turned on, and the photocharges (A signal) accumulated in the PD 201a are transferred to the FD 203 from time t2. After that, at time t10, the vertical scanning circuit 209 sets the control signal TX_A to L. The FD 203 converts the transferred charges into a potential, and this potential (A signal level) is output to the reading circuit 235 via the amplification unit 204 and the vertical output line 207. The operational amplifier 213 outputs a voltage based on the difference between the A signal level read on the vertical output line 207 and the reference voltage Vref. The inverting gain of the operational amplifier 213 is determined by the ratio of the clamp capacitor 211 and any one of the feedback capacitors 214 to 216.

Next, at time t11, the ramp signal generator 230 starts outputting the ramp signal, and at the same time, the counter 231 starts counting up from the reset state. Further, the timing generation circuit 18 sets LATEN_S to H to enable Latch_S. The comparator 221 compares the output signal of the operational amplifier 213 with the ramp signal output by the ramp signal generator 230. When the ramp signal level exceeds the output signal level of the operational amplifier 213, the output of the comparator 221 changes from L to H (time t12). When the output of the comparator 221 changes from L to H while LATEN_S is H, the Latch_S 223 stores the counter value output by the counter 231 at that time. The counter value stored in Latch_S223 corresponds to a digital value (A signal data) representing the A signal level. Since LATEN_N is L, Latch_N222 is invalid and the count value is not stored. After that, at time t13, when the ramp signal level reaches a predetermined value, the ramp signal generator 230 stops the output of the ramp signal, and the timing generation circuit sets LATEN_S to L.

After that, from time t14 to t15, the horizontal scanning circuit 232 sequentially sets the control signal hsr(h) to H for each fixed period. As a result, the switches 226 and 227 of each read circuit 235 are turned on for a certain period of time and then turned off. The N signal data and the A signal data held in the Latch_N 222 and Latch_S 223 of each read circuit 235 are read to the common output lines 228 and 229, respectively, and input to the data output unit 233. The data output unit 233 outputs the value obtained by subtracting the N signal data from the A signal data for the A signal data and the N signal data output from each reading circuit 235 to the outside.

From time t16 to t17, the vertical scanning circuit 209 sets the control signals TX_A and TX_B to H, and turns on the transfer switches 202a and 202b. As a result, photoelectric charges are transferred from both PDs 201a and 201b to the FD 203. The FD 203 converts the transferred charges into a potential, and this potential (A+B signal level) is output to the readout circuit 235 via the amplification unit 204 and the vertical output line 207. The operational amplifier 213 outputs a voltage based on the difference between the A+B signal level read to the vertical output line 207 and the reference voltage Vref.

Next, at time t18, the ramp signal generator 230 starts outputting the ramp signal, and at the same time, the counter 231 starts counting up from the reset state. Further, the timing generation circuit 18 sets LATEN_S to H to enable Latch_S. The comparator 221 compares the output signal of the operational amplifier 213 with the ramp signal output by the ramp signal generator 230. When the ramp signal level exceeds the output signal level of the operational amplifier 213, the output of the comparator 221 changes from L to H (time t19). When the output of the comparator 221 changes from L to H while LATEN_S is H, the Latch_S 223 stores the counter value output by the counter 231 at that time. The counter value stored in Latch_S223 corresponds to a digital value (A+B signal data) representing the A+B signal level. After that, at time t20, when the ramp signal level reaches a predetermined value, the ramp signal generator 230 stops the output of the ramp signal, and the timing generation circuit sets LATEN_S to L.

After that, from time t21 to t22, the horizontal scanning circuit 232 sequentially sets the control signal hsr(h) to H for each fixed period. As a result, the switches 226 and 227 of each read circuit 235 are turned on for a certain period of time and then turned off. The N signal data and the A+B signal data held in the Latch_N 222 and Latch_S 223 of each read circuit 235 are read to the common output lines 228 and 229, respectively, and input to the data output unit 233. The data output unit 233 outputs, for the A+B signal data and the N signal data output from each read circuit 235, a value obtained by subtracting the N signal data from the A+B signal data to the outside.

At time t22, the timing generation circuit 18 sets the control signal RES_C to H. At time t23, the vertical scanning circuit 209 sets the control signal RES to H, and at time t24, the vertical scanning circuit 209 sets the control signal SEL to L, and the read operation for one row is completed. By repeating this operation for a predetermined number of rows, an image signal for one screen is acquired.

In this embodiment, the camera 100 has a still image mode and a moving image mode. When the still image mode is set, the system control unit 50 controls to read out pixel data for all rows from the image sensor 14. Further, when the moving image mode is set, the system control unit 50 controls to read pixel data from the image sensor 14 at, for example, a three-row cycle (one row is read and two rows are skipped). As described above, in the present embodiment, the number of rows to be read is smaller in the moving image mode than in the still image mode. However, the reading method in the still image mode and the moving image mode is not limited to the above method.

By the reading method described with reference to FIG. 4, the A signal and the A+B signal from which the reset noise is removed can be read from the image sensor 14. The A signal is used as a focus detection signal, and the A+B signal is used as a signal forming a captured image. The A+B signal and the A signal are also used to generate a B signal for focus detection.

The image sensor 14 of the present embodiment has two types of read modes, that is, an all-pixel read mode and a thinned-out read mode. The all-pixel reading mode is a mode in which all effective pixels are read, and is set, for example, when obtaining a high-definition still image. The thinning-out reading mode is a mode in which less pixels are read out than the all-pixels reading mode, and it is necessary to read out at a high speed when obtaining pixels having a resolution lower than that of a high-definition still image such as a moving image or a preview image. If set. In the thinning-out reading mode, for example, pixels can be thinned out and read at the same ratio in both the horizontal and vertical directions so as not to change the aspect ratio of the image. Note that "thinning" includes not only not performing reading itself but also discarding (ignoring) the read signal and generating one signal from a plurality of read signals. For example, S/N can be improved by averaging signals read from a plurality of adjacent pixels to generate one signal.

FIG. 5 is a diagram illustrating a conjugate relationship between the exit pupil plane of the photographing lens and the photoelectric conversion unit and a projected image of the photoelectric conversion unit on the exit pupil plane.
FIG. 5A shows the conjugate relationship between the exit pupil plane of the taking lens 300 and the photoelectric conversion units 201a and 201b of the pixel 200 (center pixel) arranged near the center of the image plane of the image sensor 14. The photoelectric conversion units 201a and 201b in the image sensor 14 and the exit pupil plane of the taking lens 300 are designed to have a conjugate relationship by the on-chip microlens 201i. The exit pupil surface of the taking lens 300 substantially coincides with the surface on which the iris diaphragm for adjusting the light amount is provided.

The taking lens 300 is a zoom lens having a variable magnification function. Some zoom lenses change the size of the exit pupil and the distance from the image plane to the exit pupil (exit pupil distance) when the zooming operation is performed. FIG. 5A shows a state in which the focal length of the taking lens 300 is at the center between the wide-angle end and the telephoto end. With the exit pupil distance Dl in this state as a standard value, the eccentricity parameter is optimally designed according to the shape of the on-chip microlens and the image height (distance from the screen center or XY coordinates).

In FIG. 5A, the taking lens 300 includes a first lens group 101, a lens barrel member 101b holding the first lens group, a third lens group 105, and a lens barrel member 105b holding the third lens group. is doing. The photographic lens 300 also includes a diaphragm 102, an aperture plate 102a that defines the opening diameter when the diaphragm is open, and diaphragm blades 102b that adjust the opening diameter when the diaphragm is closed. The limiting members 101b, 102a, 102b, and 105b for the light fluxes CLa and CLb passing through the taking lens 300 show optical virtual images when observed from the image plane. L is the outermost part of the light beams CLa and CLb. The synthetic aperture in the vicinity of the diaphragm 102 is defined as the exit pupil of the taking lens 300, and the distance from the image plane is the exit pupil distance Dl.

The photoelectric conversion units 201a and 201b are arranged in the lowermost layer of the pixel 200. Wiring layers 201e to 201g, a color filter 201h, and an on-chip microlens 201i are provided on the upper layers of the photoelectric conversion units 201a and 201b. The photoelectric conversion units 201a and 201b are projected onto the exit pupil plane of the taking lens 300 by the on-chip microlens 201i. In other words, the exit pupil is projected onto the surfaces of the photoelectric conversion units 201a and 201b via the on-chip microlens 201i.

FIG. 5A shows projected images 501 and 502 of the photoelectric conversion units 201a and 201b on the exit pupil plane of the taking lens 300. A circle 400 on the exit pupil plane indicates the maximum incident range of the light flux on the pixel 200, which is defined by the aperture plate 102a of the diaphragm 102. Since the circle 400 is defined by the aperture plate 102a, the circle 400 is also described as 102a in FIG. Since the central pixel is shown in FIG. 5A, vignetting of the light flux is symmetric with respect to the optical axis, and the photoelectric conversion units 201a and 201b receive the light flux that has passed through the pupil regions of the same size. Since most of the projected images 501 and 502 are included in the circle 400, vignetting of the light flux hardly occurs. Therefore, when the signals photoelectrically converted by the photoelectric conversion units 201a and 201b are added, the result obtained by photoelectrically converting the light flux passing through the circle 400, that is, almost the entire exit pupil region is obtained. The area of the exit pupil received by the photoelectric conversion unit 201a is referred to as a first pupil area. The area of the exit pupil received by the photoelectric conversion unit 201b is referred to as a second pupil area. Further, a region obtained by combining the first pupil region and the second pupil region is called a third pupil region.

Crosstalk that occurs between pixels will be described with reference to the structure of the pixel 200 illustrated in FIG. Crosstalk between adjacent pixels is classified into two types depending on the factors. One is optical crosstalk generated by the arrangement of the on-chip microlens 201i, the color filter 201h, the wiring layers 201e to 201g, stray light due to reflection and scattering of transmitted light, and the like. The other is electrical crosstalk that occurs when electric charges generated in the photoelectric conversion unit of each pixel move to another pixel.

Regarding optical crosstalk, since the transmittance, reflectance, refraction, etc. differ depending on the wavelength of light, the amount of crosstalk generated (the amount of movement of light to adjacent pixels) differs for each color. In addition, the amount of generated crosstalk varies depending on the incident angle of light. Further, due to the anisotropy of the wiring layers 201e to 201g of the pixel, anisotropy also occurs in the direction in which crosstalk occurs.

Regarding the electrical crosstalk, the depth of penetration into the photoelectric conversion unit (the depth of photoelectric conversion) differs depending on the wavelength of light, so the amount of crosstalk generated (the amount of electrification movement to adjacent pixels) is Different for each color. In addition to the wavelength, the electrical crosstalk has anisotropy in the direction in which the crosstalk occurs due to the anisotropy of the wiring layers 201e to 201g of the pixels and the height of the barrier against signal charges between the pixels. Occurs. The height of the barrier against signal charges between pixels is disclosed in Patent Document 3.

In the present embodiment, it is assumed that the pixels included in the image sensor 14 have anisotropy in crosstalk and that the horizontal direction has a larger crosstalk amount than the vertical direction. Such a situation occurs, for example, when the height of the barrier of the pixel and the wiring layer are configured such that crosstalk is less likely to occur in the vertical direction than in the horizontal direction.

It will be described later that the focus detection error occurs when the crosstalk generation amount has wavelength dependency, incident angle dependency, or anisotropy. According to the image pickup apparatus of the present embodiment, highly accurate focus detection that does not depend on the occurrence of crosstalk can be realized.

FIG. 6 is a diagram for explaining the characteristics of the incident angle light receiving sensitivity distribution for each of the spectra of the photoelectric conversion unit.
FIG. 6 shows incident angle intensity distributions of the photoelectric conversion units 201a and 201b provided in the pixels 200 arranged near the center of the image plane of the image sensor 14. The photoelectric conversion units 201a and 201b are assumed to be photoelectric conversion units corresponding to pixels having red (R) and green (G) color filters. The horizontal axis is the incident angle, and the right side is positive. The vertical axis represents the signal intensity corresponding to the light receiving sensitivity, and is normalized at the intersection of the signal intensities of the photoelectric conversion units 201a and 201b. The intensity distribution of the photoelectric conversion unit 201a has the maximum intensity when the incident angle is negative, and the intensity distribution of the photoelectric conversion unit 201b has the maximum intensity when the incident angle is positive. Due to the difference in the refractive index depending on the wavelength, the shapes of the R and G incident angle light receiving sensitivities are different. The color filter applied to the pixels of the image sensor 14 includes blue (B), but it is omitted here for simplification of description. However, the intensity distribution shape of blue is different from that of red or green.

As shown in FIG. 6, the change rate of the signal intensity with respect to the incident angle is different between R and G. Therefore, in the area deviating from the angle of the intersection of the signal intensities, the signal intensity ratio of R and G differs between the photoelectric conversion units 201a and 201b. For example, when the incident angle is more negative than the intersection of the signal intensities, R has a smaller signal intensity than G in the photoelectric conversion unit 201a. On the other hand, in the photoelectric conversion unit 201b, R has a larger signal intensity than G. In this case, the paired signals used for focus detection have different R and G signal intensity ratios.

FIG. 7 is a diagram illustrating the relationship among the first pupil region, the second pupil region, and the third pupil region at the peripheral image height of the image sensor.
As described above, the area of the exit pupil received by the photoelectric conversion unit 201a is the first pupil area 501. The area of the exit pupil received by the photoelectric conversion unit 201b is the second pupil area 502. An area obtained by combining the first pupil area and the second pupil area, that is, the exit pupil 400 of the imaging optical system is the third pupil area. In FIG. 7, the horizontal direction of the exit pupil 400 is the X axis and the vertical direction is the Y axis.

In the state shown in FIG. 7A, the exit pupil distance Dl of the image pickup optical system and the set pupil distance Ds of the image pickup element are the same. In this state, the first pupil area 501 and the second pupil area 502 divide the exit pupil 400 of the imaging optical system into substantially even pupils. Considering the incident angle light receiving sensitivity distribution, the ray incident angle θ to each image height is substantially equal to the incident angle θc with the highest sensitivity of the pixel having the photoelectric conversion units 201a and 201b (θ=θc). In other words, in the incident angle light receiving sensitivity distribution shown in FIG. 6, when the incident angle θc is the origin, the light flux in a region symmetrical to the vertical axis is received at any image height, and focus detection is performed. The RGB signal strength ratios of the pair of signals used for are almost equal.

In the state shown in FIG. 7B, the exit pupil distance Dl of the image pickup optical system is shorter than the set pupil distance Ds of the image pickup element. Further, in the state shown in FIG. 7C, the exit pupil distance Dl of the image pickup optical system is longer than the set pupil distance Ds of the image pickup element. 7B and 7C, at the peripheral image height of the image pickup device, the exit pupil of the image pickup optical system shifts the entrance pupil of the image pickup device, and the exit pupil 400 of the image pickup optical system becomes The pupils are unevenly divided. Considering the incident angle light receiving sensitivity distribution, the light ray incident angle θ in the states of FIGS. 7B and 7C is calculated from the incident angle θc with the highest sensitivity of the pixel having the photoelectric conversion units 201a and 201b as the image height increases. It shifts. Therefore, the larger the deviation between the exit pupil distance of the lens unit 100 and the set pupil distance Ds and the higher the image height, the more the photoelectric conversion units 201a and 201b have the light flux of the incident angle far from the origin in the incident angle light receiving sensitivity distribution of FIG. Will be received. As a result, the difference in RGB signal strength between the pair of signals used for focus detection becomes large. On the other hand, at the center image height of the image sensor, the pupil shift does not occur regardless of the relationship between the exit pupil distance Dl of the image pickup optical system and the set pupil distance Ds of the image sensor. There is no difference in RGB signal strength between the pair of signals used. In the present embodiment, the states of FIGS. 7B and 7C can occur when the focal length of the taking lens 300 is at the wide-angle end or the telephoto end. Moreover, the same state may occur when the photographing lens is replaced.

FIG. 8 is a diagram showing an example of setting the focus detection area.
FIG. 8 shows focus detection areas 401 and 402 set within the shooting range 400. The phase difference detection AF based on the output of the image sensor 14 is performed on the focus detection area. The focus detection area 401 is a focus detection area set at a so-called central image height, in which the intersection of the optical axis of the photographing optical system and the image sensor and the center of the focus detection area coincide with each other. The focus detection area 402 is a focus detection area that is set at a so-called peripheral image height, in which the center of the focus detection area is away from the intersection of the optical axis of the photographing optical system and the image sensor. When performing focus detection using the output of the pixels of the image sensor 14, the pixels included in the area of the image sensor 14 corresponding to the focus detection areas 401 and 402 are detected in both the contrast detection method and the phase difference detection method. The output is used. Therefore, it can be said that the focus detection areas 401 and 402 are set in the image sensor 14, and in the following, the focus detection areas 401 and 402 will be described as pixel areas of the image sensor 14 for ease of description and understanding. In addition, it is assumed that the pixels 200 having the configuration shown in FIG. 2A are arranged in 4 rows and 2N columns in the focus detection areas 401 and 402. Note that this is merely an example, and the number and size (the number of pixels included) of the focus detection areas can be appropriately determined within a range that does not hinder the phase difference detection.

FIG. 9 is a diagram showing an example of pixels arranged in the focus detection area.
FIG. 9 shows pixels of 4 rows and 2N columns arranged in the focus detection areas 401 and 402. A pixel (PD 201a) used to generate a signal of an AF A image at the i-th row and the j-th column and its output are represented as A(i,j). Similarly, the pixel (PD 201b) used to generate the AF B image signal at the i-th row and the j-th column and its output are denoted as B(i,j). In the example shown in FIG. 9, pixels having red (R) and green (Gr) color filters are alternately arranged in the first and third rows of the pixel region. Pixels having green (Gb) and blue (B) color filters are alternately arranged in the second and fourth rows. In the present embodiment, the green pixels arranged next to red (R) in the left-right direction are designated as Gr, and the green pixels arranged next to blue (B) in the left-right direction are designated as Gb.

FIG. 10 is a diagram showing a line image intensity distribution regarding signals obtained from pixels in the focus detection area 401. FIG. 11 is a diagram showing a line image intensity distribution regarding signals obtained from pixels in the focus detection area 402.
10 and 11 show signal intensities for each arrangement of color filters, that is, for each color having different spectral characteristics. The horizontal axis represents the pixel number. The vertical axis represents the signal intensity corresponding to the light amount. Two graphs are vertically adjacent to each other so that the relative positional relationship between the A image and the B image can be easily understood. The graphs in FIGS. 10A and 11A show A images. The graphs of FIGS. 10B and 11B show the B image.

In FIGS. 10A and 10B, the red (R) signal is a long broken line, the green (Gr) signal is a short broken line, the green (Gb) signal is a solid line, and the blue (B) signal is It is indicated by a one-dot chain line. The two green signals (Gr, Gb) are shown as overlapping, since they give substantially equal outputs. On the other hand, the peak heights of red and blue signals are different from those of green signals due to the difference in spectral characteristics of the subject.

Further, FIGS. 10A and 10B show the signal intensity in the vicinity of the in-focus state, so that the image shift does not occur between the A image and the B image, and the centers of gravity are aligned. Since the focus detection area 401 (FIG. 8) has the central image height, the degree of vignetting by the taking lens 300 is substantially the same, and the difference between the signals of the A image and the B image is small.

In FIGS. 11A and 11B, a red (R) signal is a long dashed line, a green (Gr) signal is a short dashed line, a green (Gb) signal is a solid line, and a blue (B) signal is It is indicated by a one-dot chain line. Regarding the image A corresponding to the graph of FIG. 11A, the two green signals (Gr, Gb) are shown in an overlapping manner because substantially equal outputs are obtained. On the other hand, the peak heights of red and blue signals are different from those of green signals because of differences in spectral characteristics of the subject. In image A, the pixel position of the center of gravity of red and blue signals is further displaced. This indicates that line images are formed at different positions for each color due to the chromatic aberration of magnification of the taking lens 300.

Regarding the B image corresponding to the graph of FIG. 11B, two green signals (Gr, Gb) indicate that different outputs are obtained for both the peak height and the pixel position of the center of gravity. Despite the green signal having the same spectral characteristic, Gr and Gb have different signal outputs. This is because the Gr and Gb of the B image have a difference in output due to the influence of the signal intensity ratio of the A image and the B image, chromatic aberration of magnification of the photographing optical system, anisotropy of crosstalk between pixels, and the like. The influence of crosstalk anisotropy between pixels will be described later with reference to FIG. On the other hand, the peak heights of red and blue signals are different from those of green signals because of differences in spectral characteristics of the subject. In the B image, the positions of the red and blue signals are further displaced in the horizontal axis direction. This indicates that line images are formed at different positions for each color due to the chromatic aberration of magnification of the taking lens 300.

Further, as shown in FIGS. 11A and 11B, the output of the B image signal is smaller than the output of the A image signal in any color. This indicates that vignetting occurs asymmetrically in the light fluxes of the A image and the B image that have passed through the taking lens 300. Due to the difference in the incident angle light receiving sensitivity distributions of the A image signal and the B image signal corresponding to the incident angle of the photographing light flux, the output of the B image signal is smaller than that of the A image signal.

Further, as shown in FIGS. 11A and 11B, the intensity ratio of each color (RGB) is different between the A image and the B image. In the A image, the peak of the signal decreases in the order of G, R, and B, whereas in the B image, the signal peak decreases in the order of R, G, and B. This indicates that the sensitivity distribution for each color of the light flux incident on the pixels at the peripheral image height differs between the A image and the B image.

FIG. 12 is a diagram illustrating an example of crosstalk between pixels.
FIG. 12A shows a crosstalk generation path for the Gr pixel. FIG. 12B shows a crosstalk generation path for the Gb pixel. Although crosstalk between pixels must be considered bidirectionally, the amount of crosstalk mixed in Gr and Gb pixels will be described below in order to explain the difference between Gr and Gb. Regarding crosstalk between the A image and the B image in the same pixel, it works to cancel the image shift amount, and causes a focus detection error. However, since the light amount ratio between the A image and the B image is substantially the same in both the Gr pixel and the Gb pixel, the ratio of the crosstalk amounts is also the same, which does not cause a difference between Gr and Gb of the B image. The description is omitted.

First, the output of the Gr pixel and the Gb pixel of the A image will be described with reference to FIGS. As shown in FIG. 12A, A-Gr receives crosstalk (CT_h1) from B-R on the left side. However, since the B-R light amount is smaller than the A-Gr light amount as shown in FIG. 11, the output change due to crosstalk is small. Furthermore, A-Gr receives crosstalk (CT_v1) from A-B from above and below. However, since the light amount of A-B is smaller than that of A-Gr as shown in FIG. 11, the change in output due to crosstalk is small. That is, A-Gr is less affected by crosstalk under the shooting conditions corresponding to FIG.

As shown in FIG. 12B, A-Gb receives the crosstalk (CT_h3) from BB on the left side. However, the light amount of B-B is smaller than that of A-Gb as shown in FIG. 11B, and thus the output change due to crosstalk is small. Furthermore, A-Gb receives crosstalk (CT_v3) from A-R from above and below. The light amount of A-R is relatively larger than that of A-Gb, as shown in FIG. However, the pixels included in the image sensor 14 of the present embodiment differ in the amount of crosstalk generated in the horizontal and vertical directions, and the amount of crosstalk is smaller in the vertical direction. Therefore, A-Gb is not significantly affected by crosstalk under the shooting conditions of FIG. As a result, there is no large difference in output between A-Gr and A-Gb.

Next, with reference to FIGS. 12A and 12B, the output of the Gr pixel and the Gb pixel of the B image will be described. B-Gr receives the crosstalk (CT_h2) from A-R on the right side. As shown in FIGS. 11A and 11B, the light amount of A-R is larger than the light amount of B-Gr, and the output change due to crosstalk is large. Therefore, at the pixel position of R (horizontal axis) of the A image in FIG. 11A, B-Gr causes an output change due to crosstalk.

Further, B-Gr receives crosstalk (CT_v2) from BB from above and below. However, as shown in FIGS. 11A and 11B, the light quantity of BB is smaller than the light quantity of B-Gr, so that the output change due to crosstalk is small. That is, in the output of B-Gr, the position of the center of gravity changes due to the influence of the output of A-R under the shooting conditions of FIG.

When the crosstalk amount is large and the signal amount of the pixel which has received the crosstalk is small, the output change of the pixel becomes large. The ratio CT_h2/B-Gr between the signal of B-Gr and the crosstalk amount (CT_h2) leaked from the A-R to the B-Gr pixel is set as the first ratio. Further, the ratio CT_h4/B-Gb between the signal of B-Gb and the amount of crosstalk (CT_h4) leaking from the pixel AB to the pixel B-Gb is set as the second ratio. The larger each of these ratios, the greater the change in the position of the center of gravity of the pixel corresponding to the denominator of the ratio due to the influence of crosstalk. When the first ratio is larger than the second ratio, B-Gr is larger than B-Gb with respect to the influence of crosstalk. Therefore, in this case, B-Gb is weighted with a weight larger than B-Gr and focus detection is performed to reduce the change in the position of the center of gravity due to crosstalk, and then the image shift between the A image and the B image is performed. The amount (center of gravity interval) can be calculated. Although the present embodiment has been described focusing on B-Gr and B-Gb, the same applies to other different pixels.

Next, the output of the Gb pixel of the B image will be described. As shown in FIG. 12B, B-Gb receives the crosstalk (CT_h4) from AB on the right side. However, as shown in FIGS. 11A and 11B, the light quantity of AB is smaller than the light quantity of B-Gb, so that the output change due to crosstalk is small.

In addition, B-Gb receives crosstalk (CT_v4) from R of B image from above and below. As shown in FIG. 11B, the light amount of B-R is relatively larger than the light amount of B-Gb. However, the pixels of the image sensor 14 of the present embodiment differ in the amount of crosstalk generated in the horizontal and vertical directions, and the amount of crosstalk is smaller in the vertical direction. Therefore, B-Gb is less affected by crosstalk under the shooting conditions of FIG. As a result, mainly due to crosstalk from A-R, the output of B-Gr becomes larger than that of B-Gb, and the position of the center of gravity shifts to the left.

As described above, despite the fact that defocusing has not been performed, due to the influence of crosstalk, the center of gravity of the line image intensity distribution of Gr is on the left side with respect to Gb, as shown in FIG. 11(B). Deviated. The focus detection corresponds to the calculation of the image shift amount (the center-of-gravity interval) between the A image and the B image, and therefore the gravity center position error of Gr becomes a focus detection error. In the present embodiment, the system control unit 50 calculates the correlation amount of the first spectral characteristic that is less affected by crosstalk and the correlation amount of the second spectral characteristic that does not consider the influence of crosstalk, and determines the focus detection area Weighting addition of the correlation amount is performed according to the image height. The weighted addition is to add weights to the correlation amount of the first spectral characteristic and the correlation amount of the second spectral characteristic. Specifically, according to the control of the system control unit 50, the AF unit 42 controls the magnitude of the weight for the first correlation amount and the second correlation amount according to the image height of the focus detection area. This makes it possible to suppress focus detection error and realize highly accurate focus detection. Focus detection methods such as calculation of the correlation amount and weighted addition will be described later.

The following factors are conceivable as factors that cause different outputs from the pixels of Gr and Gb that should originally obtain the same output.
・Spectroscopic characteristics of subject (RGB ratio)
・Angle dependence of incident angle sensitivity distribution of paired photoelectric converters, spectral dependence ・Image height of shooting optical system and focus detection area ・Chromatic aberration of magnification caused by shooting optical system ・Amount of crosstalk between pixels (each adjacent direction)

Different outputs can be obtained from the Gr pixel and the Gb pixel depending on the above factors. Therefore, it is difficult to obtain the contribution of each factor and the influence of the obtained focus detection error for each factor. Further, since the spectral characteristic of the subject also contributes to the focus detection result, it is difficult for the imaging device to store the crosstalk amount and the focus detection error as correction values in advance. Therefore, in the present embodiment, when the image height of the focus detection area includes a peripheral image height region where crosstalk is likely to occur, the system control unit 50 has a correlation amount of the first spectral characteristic that is less affected by crosstalk. Is set to be larger than the weight of the correlation amount of the second spectral characteristic. Then, the system control unit 50 performs focus detection using the correlation amount obtained by weighting and adding the correlation amount of the first spectral characteristic and the correlation amount of the second spectral characteristic.

(Focus detection operation)
FIG. 13 is a flowchart illustrating an example of the focus adjustment operation by the image pickup apparatus.
The process described with reference to FIG. 13 is performed when the main mirror 130 and the sub mirror 131 are retracted (mirror up) out of the optical path, specifically, during live view display (during video recording for display) or video recording. This is executed in (when recording a moving image for recording). Note that, although FIG. 13 illustrates the automatic focus detection of the phase difference detection method using the output of the image sensor 14, the automatic focus detection of the contrast detection method can also be performed. Further, the processing described with reference to FIG. 13 is executed under the control of the system control unit 50.

In step S501, the system control unit 50 determines whether a focus detection start instruction has been input. The system control unit 50 determines that the focus detection start instruction is input, for example, when the SW1 (shutter switch 62) is turned on or when the user inputs an operation using the operation unit 70. If the focus detection start instruction has not been input, the process returns to S501. If the focus detection start instruction is input, the process proceeds to S502. It should be noted that the system control unit 50 may not only input the focus detection start instruction, but may proceed to S502 by using the start of live view display or moving image recording as a trigger.

In step S502, the system control unit 50 acquires various lens information such as the lens frame information of the taking lens 300 and the focus lens position from the lens system control unit 346 via the interface units 38 and 338 and the connectors 122 and 322. In step S<b>503, the system control unit 50 performs image processing so as to generate an AF image signal pair (focus detection RGB signal) from the pixel data in the focus detection area of the sequentially read frame image data. Instruct the section 20. The image processing unit 20 generates an image signal pair for AF and supplies it to the AF unit 42. The AF unit 42 performs a process of correcting the difference in signal level for the AF image signal pair. The AF unit 42 also detects the peak value (maximum value) and bottom value (minimum value) of the AF image signal. The image processing unit 20 processes as an image signal pair for AF for each color filter type (R, Gr, Gb, B) of the pixel to generate an image signal pair.

In S504, the image processing unit 20 generates the A signal having the first spectral characteristic and the A signal having the second spectral characteristic based on the image signal pair generated in S503 under the control of the system control unit 50. In the present embodiment, the image processing unit 20 weights the first spectral characteristic so that the influence of crosstalk is reduced. Further, the image processing unit 20 weights the second spectral characteristic so as to reduce the noise of the signal without considering the influence of crosstalk. The image processing unit 20 generates a Gb signal using only Gb for the first spectral characteristic of the color filter types (R, Gr, Gb, B) of the pixel, for example. For the second spectral characteristic, the image processing unit 20 uses all of R, Gr, Gb, and B to generate a Y signal having no characteristic for each color. At this time, the signal having the first spectral characteristic (Gb) is less susceptible to the crosstalk than the signal having the second spectral characteristic (Y). Further, the wavelength band of the first spectral characteristic is narrower than the wavelength band of the second spectral characteristic.

In the present embodiment, pixels of 4 rows and 2N columns are arranged in the focus detection areas 401 and 402 as shown in FIG. An A signal of the first spectral characteristic is generated from the A signals of the first row and the second row, and an A signal of the second spectral characteristic is generated from the A signals of the third row and the fourth row. As a result, the signals are added every two pixels in the horizontal and vertical directions, and compressed into the A signal of 2 rows and N columns. At this time, the first line shows the A signal of the first spectral characteristic (Gb), and the second line shows the A signal of the second spectral characteristic (Y). That is, the image processing unit 20 acquires the signal of the first spectral characteristic based on the output of the pixel of the first area, and the signal of the second spectral characteristic based on the output of the pixel of the second area. To get. Details of the signal addition method will be described later. Further, in this example, the signal is generated so that the first row becomes the A signal having the first spectral characteristic and the second row becomes the A signal having the second spectral characteristic. The signals may be added so as to generate the A signal or the A signal having the second spectral characteristic in both rows.

In S505, the image processing unit 20 generates a B signal with the first spectral characteristic and a B signal with the second spectral characteristic by the same processing as in S504. For example, for a focus detection area in which 4 rows of 2N-column pixels are arranged, a B signal of the first spectral characteristic (Gb) is generated from the B signals of the 1st row and the 2nd row, and the 3rd row is generated. The B signal of the second spectral characteristic (Y) is generated from the B signal of the fourth row. As a result, signals are added every two pixels in the horizontal and vertical directions, and compressed into B signals in 2 rows and N columns. At this time, the first line is the B signal of the first spectral characteristic (Gb), and the second line is the B signal of the second spectral characteristic (Y). By the processing of S504 and S505, the signal of the first spectral characteristic and the signal of the second spectral characteristic are acquired from each of the first photoelectric conversion unit and the second photoelectric conversion unit.

In step S506, the AF unit 42 determines the first correlation amount COR(k) based on the A signal having the first spectral characteristic generated in step S504 and the B signal having the first spectral characteristic generated in step S505. To calculate. In step S507, the AF unit 42 causes the second correlation amount COR( based on the A signal of the second spectral characteristic generated in step S504 and the B signal of the second spectral characteristic generated in step S505. k) is calculated. Subsequently, in S508, the AF unit 42 acquires image height information of the focus detection area. The image height information of the focus detection area acquired by the AF unit 42 is used when calculating the third correlation amount COR(k) in S509.

Next, in S509, the AF unit 42 weights the first correlation amount and the second correlation amount, and calculates the third correlation amount. With respect to the first weight given to the first correlation amount and the second weight given to the second correlation amount, the AF unit 42 causes the image height of the focus detection region to cause crosstalk. When the peripheral image height is easy to perform, the first weight is set to be larger than the second weight. Specifically, when the focus detection area includes only the image height equal to or less than the threshold value, the AF unit 42 sets, for example, the first weight:the second weight=0:1. Further, when the focus detection area includes an image height equal to or larger than the threshold value, the AF unit 42 sets, for example, the first weight:the second weight=1:0. Further, the AF unit 42 may change the weight according to the image height of the focus detection area without providing a threshold value regarding the image height. The first correlation amount according to the first spectral characteristic is generated from the image signal pair of the first and second rows of the focus detection regions 401 and 402, and the second correlation signal is generated from the image signal pair of the third and fourth rows. A second correlation amount is generated according to the spectral characteristic of. Based on the first correlation amount and the second correlation amount, the third correlation amount for one row is calculated by the processing in S509.

Next, in S510, the AF unit 42 detects the shift amount k that minimizes the third correlation amount COR(k) calculated in S509 as the image shift amount (phase difference). Then, the AF unit 42 converts the detected image shift amount into a defocus amount. The AF unit 42 outputs the defocus amount to the system control unit 50. Then, in S511, the system control unit 50 determines the focus lens drive amount and drive direction of the taking lens 300 based on the defocus amount obtained from the AF unit 42 in S510.

In step S512, the system control unit 50 transmits the focus lens drive amount and drive direction information to the lens system control unit 346 of the taking lens 300 via the interface units 38 and 338 and the connectors 122 and 322. The lens system control unit 346 transmits the focus lens drive amount and drive direction information to the focus control unit 342. The focus control unit 342 drives the focus lens based on the received information on the lens driving amount and the driving direction. As a result, the focus of the taking lens 300 is adjusted. The focus adjustment operation may be continuously executed even when the moving image data of the next frame is read. Further, the information about the focus lens drive amount and the drive direction may be directly transmitted from the system control unit 50 to the focus control unit 342.

(Subroutine of A signal generation of the first spectral characteristic and the second spectral characteristic)
FIG. 14 is a flowchart illustrating the processing of S504 and S504 of FIG.
FIG. 14A shows the process of S504 of FIG. The image processing unit 20 generates an A signal having a first spectral characteristic for N rows and an A signal having a second spectral characteristic for N rows from the A signals for 4N rows. As shown in FIG. 9, in the A signal for 4N rows, rows in which pixels having R and Gr color filters are alternately arranged and pixels having Gb and B color filters are alternately arranged. An image sensor in which rows are arranged alternately is assumed.

In S5041 of FIG. 14A, the image processing unit 20 reads the A signal of the 4N+1th row and the A signal of the 4N+2th row from the memory 30. In step S5042, the image processing unit 20 pixel-adds the A signal of the 4N+1th row and the A signal of the 4N+2th row by two rows and two columns so that the A signal has the first spectral characteristic. The image processing unit 20 weights so that R:Gr:B:Gb=0:0:0:1, for example, and performs addition to generate a Gb signal. Further, the ratio of R:Gr:B:Gb at this time can be set to an arbitrary ratio as long as it is weighted so that the influence of crosstalk is reduced.

Next, in S5043, the image processing unit 20 reads out the A signal of the 4N+3th row and the A signal of the 4N+4th row from the memory 30. In step S5044, the image processing unit 20 pixel-adds the A signal on the 4N+3th row and the A signal on the 4N+4th row by 2 rows and 2 columns so that the A signal has the first spectral characteristic. The image processing unit 20 performs weighting so that R:Gr:B:Gb=1:1:1:1, for example, and generates a Y signal that does not have characteristics for each color. In S5045, the system control unit 50 determines whether or not the processing of all 4N rows has been completed. If the process is not completed, the process returns to S5041. When the processing of all 4N rows is completed, the process proceeds to S505 of FIG.

(Sub-routine of B signal generation of the first spectral characteristic and the second spectral characteristic)
FIG. 14B shows the processing of S505 of FIG. In step S5051, the image processing unit 20 reads the B signals of the 4N+1th row and the 4N+2th row from the memory 30. Subsequently, in S5052, the image processing unit 20 generates the B signal (Gb) of the Nth row. In S5053, the image processing unit 20 reads the B signals of the 4N+3th row and the 4N+4th row from the memory 30. Subsequently, in S5054, the image processing unit 20 generates the B signal (Y) of the Nth row. S5055 is the same as S5045 in FIG.

(Subroutine for calculating the correlation amount)
15 and 16 are diagrams for explaining the subroutine for calculating the correlation amount.
FIG. 15A is a flowchart illustrating the calculation processing of the first correlation amount in S506 of FIG.
In step S5061, the AF unit 42 determines the first correlation amount based on the A signal of the first spectral characteristic (Gb) of the Nth row and the B signal of the first spectral characteristic (Gb) of the Nth row. Calculate COR(k). The method of calculating the correlation amount COR(k) is as follows. In the focus detection using the phase difference detection method, the AF unit 42 generates a pair of images having portions corresponding to the same subject, and detects the phase difference (shift amount) between the pair of images. The AF unit 42 converts the phase difference into a defocus amount and a direction. A signal sequence (A image) based on the first signal obtained from the PDs 201a of the plurality of pixels 200 existing in a predetermined direction (for example, horizontal direction) and a signal sequence (B image) based on the second signal obtained from the PD 201b. Corresponds to an image of the same subject viewed from different viewpoints. Therefore, the AF unit 42 detects the phase difference between the A image and the B image and converts it into the defocus amount and the direction, whereby the focus detection of the phase difference detection method can be realized. Then, the AF unit 42 calculates a value (correlation amount) representing the correlation between the A image and the B image at each position while changing the relative distance (shift amount) between the A image and the B image in the predetermined direction. , The shift amount with the highest correlation can be detected as the phase difference between the A image and the B image. The correlation amount is, for example, a difference cumulative value of corresponding signal values, but another value may be used as the correlation amount.

For example, A(1,1) to A(1,N) are generated as the A image, B(1,1) to B(1,N) are generated as the B image, and the shift amount k is -kmax≦ in pixel units. It is assumed that the value is changed within the range of k≦kmax. The correlation amount COR(k) at each relative position can be calculated by the following equation (1).

Next, in S5062, the AF unit 42 determines whether or not the processing of all N rows has been completed. If there is a line that has not been processed, the process returns to S5061. If all the N rows have been processed, the process proceeds to S507 in FIG.

FIG. 15B is a flowchart illustrating the calculation processing of the second correlation amount in S507 of FIG.
In S5071, the AF unit 42, based on the A signal of the second spectral characteristic (Y) of the Nth row and the B signal of the second spectral characteristic (Y) of the Nth row, outputs the second correlation amount COR( k) is calculated. The method of calculating the second correlation amount COR(k) in S5071 is the same as the method of calculating the first correlation amount COR(k) in S5061 of FIG. Subsequently, in S5072, the AF unit 42 determines whether or not the processing of all N rows has been completed. If there is a line that has not been processed, the process returns to S5071. If all the N rows have been processed, the process proceeds to S508 in FIG.

FIG. 16 is a flowchart illustrating the calculation processing of the third correlation amount in S509 of FIG.
In step S5091, the AF unit 42 determines whether the focus detection area includes an image height equal to or higher than a threshold based on the focus detection area information acquired in step S508 of FIG. If the focus detection area includes an image height equal to or higher than the threshold value, the process proceeds to S5092. If the focus detection area does not include an image height equal to or higher than the threshold value, the process proceeds to S5093.

In S5092, the AF unit 42 makes the first weight larger than the second weight. For example, the AF unit 42 sets the first weight:the second weight=1:0. The influence of crosstalk is expected when the peripheral image height is included in the focus detection area. Therefore, the AF unit 42 sets the first weight:the second weight=1:0 so that the focus detection is performed using only the signal of the first spectral characteristic that is less affected by the crosstalk. To do. After the processing of S5092, the processing proceeds to S5094.

In S5093, the AF unit 42 makes the second weight larger than the first weight. For example, the AF unit 42 sets the first weight:the second weight=0:1. If the SN ratio is not good and it is preferable to use the signals of all rows for focus detection, the AF unit 42 sets the first weight:the second weight=1:1 in step S5093. By doing so, the first weight and the second weight may be set to the same size. This setting is intended to reduce noise by adding all read signals.

Next, in step S5094, the AF unit 42 weights and adds the first correlation amount of the Nth row and the second correlation amount of the Nth row to obtain the third correlation amount of the Nth row. To calculate. At the time of weighting, the AF unit 42 gives a first weight to the first correlation amount. The AF unit 42 also gives a second weight to the second correlation amount.

Next, in S5095, the system control unit 50 determines whether or not the processing of all N rows has been completed. If there is a row for which the processing has not been completed, the processing returns to S5094. When the processing for all N rows is completed, the processing proceeds to S510 in FIG. The AF unit 42 calculates the reliability of the third correlation amount using a known method, and according to the calculated reliability, the first weight or the second weight, Alternatively, the ratio between the first weight and the second weight may be changed.

(Defocus amount calculation subroutine)
FIG. 17 is a flowchart illustrating the defocus amount calculation process in S510 of FIG.
First, in S5101, the AF unit 42 reads the third correlation amount calculated in S509 of FIG. 13 from the memory 30. Subsequently, in step S5102, the AF unit 42 executes the row addition determination of the correlation amount. When there are a plurality of rows in the focus detection area and the correlation calculation is performed, the AF unit 42 determines to perform row addition of the correlation amount. However, depending on the row for which the correlation calculation is performed, there is a row for which a highly reliable focus detection result cannot be obtained due to saturation or the like. Therefore, in the row addition determination of the correlation amount, the AF unit 42 determines whether to add the obtained third correlation amount of the Nth row to the previously added third correlation amount. Specifically, when the reliability of the third correlation amount read out in S5102 is equal to or higher than the threshold value, the AF unit 42 determines to add the third correlation amount. When the reliability of the third correlation amount is less than the threshold value, the AF unit 42 determines not to perform the row addition of the third correlation amount.

In S5103, the AF unit 42 confirms the determination result in S5102. Then, if the determination result indicates that the row addition of the correlation amount is performed, the process proceeds to S5104. If the determination result shows that the row addition of the correlation amount is not performed, the process proceeds to S5105. In S5104, the AF unit 42 adds the third correlation amount of the Nth row obtained in S5101 to the addition result of the third correlation amount obtained in advance.

Next, in step S5105, the AF unit 42 determines whether processing has been completed for all rows in the focus detection area (whether correlation calculation has been performed). If there is a line that has not been processed, the process returns to S5101. If the processing has been completed for all the rows in the focus detection area, the processing proceeds to S5106.

In S5106, the AF unit 42 calculates the defocus amount based on the third correlation amount COR(k) obtained by the processes of S5101 to S5105. First, the AF unit 42 obtains the value of the shift amount k that minimizes COR(k). Here, according to the above-mentioned formula (1), the calculated shift amount k is an integer, but in order to improve the resolution, the finally obtained shift amount k is a real number. For example, when the minimum value obtained by the equation (1) is COR(a), the AF unit 42 performs interpolation calculation from COR(a-1), COR(a), and COR(a+1), Find the real-valued shift amount that minimizes the correlation amount in the interval.

In the present embodiment, the AF unit 42 calculates the shift amount dk at which the sign of the difference value of the correlation amount COR changes as the shift amount k at which the correlation amount COR1(k) becomes the minimum. First, the AF unit 42 calculates the difference value DCOR of the correlation amount based on the following equation (2).
DCOR(k)=COR1(k)-COR1(k-1)... Formula (2)
Then, the AF unit 42 uses the difference value DCOR of the correlation amount to obtain the shift amount dk in which the sign of the difference amount changes. Assuming that the value of k immediately before the sign of the difference amount changes is k1 and the value of k when the sign changes is k2 (k2=k1+1), the AF unit 42 calculates the shift amount dk based on the following equation (3). To calculate.
dk=k1+|DCOR(k1)|/|DCOR(k1)-DCOR(k2)|
...Formula (3)

As described above, the AF unit 42 calculates the shift amount dk that maximizes the correlation amount between the A image and the B image in subpixel units. The method for calculating the phase difference between the two one-dimensional image signals is not limited to the method described above, and any known method can be used.

The AF unit 42 converts the calculated shift amount dk into a defocus amount Def by multiplying it by a predetermined defocus conversion coefficient. The defocus conversion coefficient can be obtained from optical conditions at the time of shooting (aperture, exit pupil distance, lens frame information, etc.), the image height of the focus detection area, the sampling pitch of the signals forming the A image, and the B image. . When the calculation of the defocus amount in S5106 is completed, the subroutine of the defocus amount calculation process ends, and the process proceeds to S511 in FIG.

The image pickup apparatus according to the present embodiment sets the image height of the focus detection region in relation to the correlation amount of the first spectral characteristic that is less affected by crosstalk and the correlation amount of the second spectral characteristic that does not consider the influence of crosstalk. Focus detection processing is executed based on the result of weighting and adding accordingly. As a result, even when the signal includes an error due to the influence of crosstalk or crosstalk correction, it is possible to obtain a focus detection result with high reliability regarding detection accuracy.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

50 system control unit 100 camera 300 shooting lens

Claims (12)

An image pickup device having a plurality of photoelectric conversion units that receive light fluxes that pass through mutually different pupil regions of the image pickup optical system,
The first photoelectric conversion unit that receives the light flux that passes through the first pupil region and the second photoelectric conversion unit that receives the light flux that passes through the second pupil region respectively have a signal of the first spectral characteristic. Acquisition means for acquiring a signal having the second spectral characteristic,
The first correlation amount calculated based on the signal of the first spectral characteristic and the second correlation amount calculated based on the signal of the second spectral characteristic are weighted to obtain the third correlation amount. And a control means for calculating the correlation amount,
The image pickup apparatus, wherein the control unit controls the magnitude of the weight for the first correlation amount and the second correlation amount according to the image height of the focus detection area. The acquisition means is
The first signal having the first spectral characteristic and the second signal having the second spectral characteristic are acquired from the first photoelectric conversion unit, and the first signal having the first spectral characteristic is acquired from the second photoelectric conversion unit. Acquiring a third signal of the spectral characteristic and a fourth signal of the second spectral characteristic,
The control means calculates the first correlation amount based on the first signal and the third signal, and the second correlation amount based on the second signal and the fourth signal. The image pickup apparatus according to claim 1, wherein a correlation amount of 1 is calculated. When the image height of the focus detection area includes an image height equal to or more than a threshold value, the control unit sets a first weight, which is a weight for the first correlation amount, to a weight for the second correlation amount. The image pickup device according to claim 1, wherein the image pickup device has a weight greater than the second weight. The control unit calculates the third correlation amount by using only the first correlation amount when the image height of the focus detection region includes an image height equal to or more than a threshold value. Item 3. The imaging device according to item 3. The control unit sets the second weight to be larger than the first weight when the image height of the focus detection area does not include an image height equal to or more than a threshold value. The imaging device according to item 4. The image pickup apparatus according to claim 1, wherein the control unit executes a focus detection process based on the third correlation amount. 7. The control unit changes the weights for the first correlation amount and the second correlation amount according to the reliability of the third correlation amount. The imaging device according to the item. The imaging device according to any one of claims 1 to 7, wherein the wavelength band of the first spectral characteristic is narrower than the wavelength band of the second spectral characteristic. The image pickup device according to claim 1, wherein the image pickup device includes a pixel having the plurality of photoelectric conversion units with respect to one microlens. The pixel has a color filter corresponding to any of a plurality of colors having different spectral characteristics,
The acquisition unit acquires the signal of the first spectral characteristic and the signal of the second spectral characteristic by weighting and adding outputs of the plurality of pixels. The imaging device according to. The acquisition unit acquires the signal of the first spectral characteristic based on the output of the pixel of the first area, and acquires the signal of the second spectral characteristic based on the output of the pixel of the second area. The image capturing apparatus according to claim 10, wherein the image capturing apparatus acquires. A control method for an image pickup apparatus having an image pickup device having a plurality of photoelectric conversion units for receiving light fluxes passing through different pupil regions of an image pickup optical system,
The first photoelectric conversion unit that receives the light flux that passes through the first pupil region and the second photoelectric conversion unit that receives the light flux that passes through the second pupil region respectively have a signal of the first spectral characteristic. An acquisition step of acquiring a signal of the second spectral characteristic,
The first correlation amount calculated based on the signal of the first spectral characteristic and the second correlation amount calculated based on the signal of the second spectral characteristic are weighted to obtain the third correlation amount. And a control step of calculating a correlation amount,
In the control step, the weight of the first correlation amount and the second correlation amount is controlled according to the image height of the focus detection area.

Images