JP6906360B2 - Focus detector and its control method - Google Patents

Description

The present invention relates to a focus detection device and a control method thereof.

Patent Document 1 discloses an apparatus for performing pupil division type focus detection using an image sensor in which a microlens is formed in each of two-dimensionally arranged pixels. This device has a configuration in which one microlens is shared by two photoelectric conversion units. As a result, of the two photoelectric conversion units sharing the microlens, the first photoelectric conversion unit obtains a signal based on the luminous flux emitted from the first region in the exit pupil of the photographing optical system. Further, in the second photoelectric conversion unit, a signal based on the luminous flux emitted from the second region in the exit pupil of the photographing optical system can be obtained. The phase difference (phase shift amount) of the signal trains is calculated by performing a correlation calculation between the signal trains obtained from the plurality of first photoelectric conversion units and the signal trains obtained from the plurality of second photoelectric conversion units, and the phase difference is calculated. The amount of defocus can be calculated from.

Further, by adding the outputs of the first photoelectric conversion unit and the second photoelectric conversion unit that share the microlens, it is possible to obtain the same output as a general pixel having one photoelectric conversion unit per microlens. can. Therefore, from one pixel, the output of the first photoelectric conversion unit (A signal), the output of the second photoelectric conversion unit (B signal), and the additional output of the first photoelectric conversion unit and the second photoelectric conversion unit. Three types of outputs (A + B signal) can be obtained. In Patent Document 1, the A + B signal is read after reading the output (for example, A signal) of one of the photoelectric conversion units, and the B signal is not read individually but is generated by subtracting the A signal from the A + B signal. .. Thereby, three kinds of signals can be acquired by reading out twice.

Japanese Unexamined Patent Publication No. 2014-182360

For example, even if the A signal and the B signal are read and the A + B signal is not read and the A signal and the B signal are added and generated, it is possible to acquire three types of signals by reading twice. However, the A signal and the B signal each include random noise caused by the read circuit. Therefore, the added random noise is included in the A + B signal obtained by adding the A signal and the B signal. Since an increase in random noise lowers the image quality, in Patent Document 1, the A + B signal is read out and acquired, and the A (or B) signal is subtracted from the A + B signal to generate the B (or A) signal.

When focus detection is performed, the A signal string for correlation calculation is generated from the A signal, and the B signal string for correlation calculation is generated from the B signal generated by subtracting the A signal from the A + B signal, thereby performing focus detection. Generate an image signal pair. Then, the correlation amount is calculated while changing the relative shift amount between the A signal string and the B signal string, and the shift amount that minimizes the correlation amount (maximizes the correlation) is searched for. However, since the B signal generated by subtracting the A signal from the A + B signal contains a noise component having a correlation with the noise component contained in the A signal, when the shift amount between the A signal string and the B signal string is 0, the other The magnitude of the noise component has a peculiar value with respect to the shift amount of. Then, in a state where the SN ratio is low, for example, when the contrast or brightness of the subject is low, the focus detection accuracy may decrease due to the noise component when the shift amount is 0.

Further, the noise components between the signal sequences constituting the image signal pair have a correlation only when the B signal is generated by subtracting the A signal from the A + B signal. For example, when the signals read from the first photoelectric conversion unit and the second photoelectric conversion unit are amplified by the same amplifier, and when the noise source in the signal path is shared, the signal strings constituting the image signal pair are also formed. The noise components in between have a correlation.

The present invention has been made in view of such problems of the prior art. The present invention relates to a focus detection device that performs phase difference detection type focus detection based on an image signal pair obtained from an image sensor and a control method thereof, and affects the influence of correlated noise contained in the image signal pair on focus detection. The purpose is to suppress it.

The above-mentioned purpose is to obtain a plurality of first signals obtained from a plurality of first photoelectric conversion units that receive a light beam passing through the first pupil region of the ejection pupil of the photographing optical system, and the ejection pupil of the photographing optical system. A generation means for generating a plurality of image signal pairs from a plurality of second signals obtained from a plurality of second photoelectric conversion units that receive a light beam passing through the second pupil region, and a plurality of image signal pairs. For each, a calculation means for calculating the phase difference of the pair of image signals by correlation calculation using a pair of image signals constituting the image signal pair, and a photographing optical system based on the phase difference calculated by the calculation means. A focus detection device having an adjusting means for adjusting the focusing distance, each of a plurality of image signal pairs is composed of a first image signal and a second image signal, and the generating means is a first image signal. A first image signal is generated from the signal, and a second image signal is generated from the second signal. Among the plurality of image signal pairs, the first image signal and the second image forming the first image signal pair. The correlation of the noise component contained in the first image signal and the second image signal constituting the second image signal pair is lower than the correlation of the noise component contained in the signal, and the adjusting means is the contrast of the subject. when the evaluation value of is greater than or equal to the threshold based on the phase difference between the first image signal to, if the evaluation value is less than the threshold value based on the phase difference of the second image signal to the photographing optical system It is achieved by a focus detector characterized by adjusting the focusing distance of the.

According to the present invention, in a focus detection device that performs phase difference detection type focus detection based on an image signal pair obtained from an image sensor and a control method thereof, correlated noise contained in the image signal pair gives focus detection. It is possible to suppress the influence.

A block diagram showing a functional configuration example of a camera system as an example of an imaging device including a focus detection device according to an embodiment. The figure which shows the structural example of the image sensor in embodiment The figure which shows the structural example of the image sensor in embodiment Timing chart showing an operation example of the image sensor of FIG. The figure which shows the relationship example of the photoelectric conversion region and an exit pupil in an embodiment. The figure which shows the example of the photographing range and the focus detection area in embodiment. The figure which shows typically the example of the image signal pair generation method in 1st Embodiment. Flow chart showing an example of the focus adjustment operation in the embodiment A flowchart showing an example of a method of calculating the defocus amount in the embodiment. The figure which shows typically the example of the image signal pair generation method in 2nd Embodiment. The figure which shows typically the influence which the noise component which has a correlation has on the calculation of the correlation amount.

● (First embodiment)
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. Here, an embodiment in which the focus detection device according to the present invention is applied to an interchangeable lens type digital single-lens reflex camera (camera system) will be described. However, the focus detection device according to the present invention can be applied to any electronic device having an image pickup device capable of generating a signal used for focus detection of a phase difference detection method. Such electronic devices include not only cameras such as digital still cameras and digital video cameras in general, but also mobile phones having camera functions, computer devices, media players, robot devices, game devices, home appliances, and the like. Not limited to.

FIG. 1 is a diagram showing a configuration example of a camera system including a camera having an interchangeable shooting lens and a shooting lens as an example of an imaging device including the focus detection device according to the embodiment of the present invention. In FIG. 1, the camera system comprises a camera 100 and an interchangeable photographing lens 300.

The luminous flux that has passed through the photographing lens 300 passes through the lens mount 106, is reflected upward by the main mirror 130, and is incident on the optical viewfinder 104. The optical viewfinder 104 allows the photographer to take a picture while observing the optical image of the subject. In the optical viewfinder 104, some functions of the display unit 54, for example, a focus display, a camera shake warning display, an aperture value display, an exposure compensation display, and the like are installed.

A part of the main mirror 130 is composed of a semitransparent half mirror, and a part of the light flux incident on the main mirror 130 passes through this half mirror part and is reflected downward by the sub mirror 131 to the focus detection device 105. Incident. The focus detection device 105 is a phase difference detection type focus detection device having a secondary imaging optical system and a line sensor, and outputs a pair of image signals to the AF unit (autofocus unit) 42. The AF unit 42 performs a phase difference detection calculation on a pair of image signals to obtain the amount and direction of defocus of the photographing lens 300. Based on this calculation result, the system control unit 50 controls the drive of the focus lens to the focus control unit 342 (described later) of the photographing lens 300.

When the focus adjustment process of the photographing lens 300 is completed and still image shooting is performed, electronic viewfinder display is performed, or moving image shooting is performed, the main mirror 130 and the sub mirror 131 are moved out of the optical path by a quick return mechanism (not shown). Evacuate to. Then, the luminous flux that has passed through the photographing lens 300 and is incident on the camera 100 can be incident on the image sensor 14 via the shutter 12 for controlling the exposure amount. After the photographing operation by the image sensor 14 is completed, the main mirror 130 and the sub mirror 131 return to the positions as shown in the drawing.

The image pickup device 14 is a CCD or CMOS image sensor, and has a configuration in which a plurality of pixels having a photoelectric conversion region (or photodiode) are arranged two-dimensionally. The image sensor 14 outputs an electric signal corresponding to the optical image of the subject. The electrical signal photoelectrically converted by the image sensor 14 is sent to the A / D converter 16, and the analog signal output is converted into a digital signal (image data). As will be described later, the A / D converter 16 may be incorporated in the image sensor 14.

The image pickup device 14 according to the present embodiment is configured such that at least a part of the pixels has a plurality of photoelectric conversion regions (or photodiodes). As described above, the pixel having such a configuration can output the signal used for the focus detection of the phase difference detection method. Therefore, even when the main mirror 130 and the sub mirror 131 are retracted out of the optical path by the quick return mechanism and no light is incident on the focus detection device 105, the focus detection of the phase difference detection method using the output of the image sensor 14 can be performed. It is possible.

The timing generation circuit 18 supplies a clock signal and a control signal to the image sensor 14, the A / D converter 16, and the D / A converter 26. The timing generation circuit 18 is controlled by the memory control unit 22 and the system control unit 50. The system control unit 50 determines the control signal for reading the output of a part of the photoelectric conversion region or adding and reading the output of all the photoelectric conversion regions from the pixel having the plurality of photoelectric conversion regions by the timing generation circuit 18 Is controlled and supplied to the image sensor 14.

The image processing unit 20 applies predetermined processing such as pixel interpolation processing, white balance adjustment processing, and color conversion processing to the image data from the A / D converter 16 or the image data from the memory control unit 22.

The image processing unit 20 is also used for the focus detection of the phase difference detection method from the output signal used for generating the focus detection signal among the image data (output signal of the image sensor 14) from the A / D converter 16. Generate a pair of signal sequences. After that, the pair of signal trains is sent to the AF unit 42 via the system control unit 50. The AF unit 42 detects the amount of phase shift (shift amount) between the signal trains by the correlation calculation of the pair of signal trains, and converts the amount of phase shift into the defocus amount and the defocus direction of the photographing lens 300. The AF unit 42 outputs the converted amount and direction of defocus to the system control unit 50. The system control unit 50 drives the focus lens through the focus control unit 342 of the photographing lens 300 and adjusts the focusing distance of the photographing lens 300.

Further, the image processing unit 20 can calculate the contrast evaluation value based on the signal (corresponding to the above-mentioned A + B signal) for generating normal image data obtained from the image sensor 14. The system control unit 50 takes an image with the image sensor 14 while changing the focus lens position through the focus control unit 342 of the photographing lens 300, and examines the change in the contrast evaluation value calculated by the image processing unit 20. Then, the system control unit 50 drives the focus lens to a position where the contrast evaluation value is maximized. As described above, the camera 100 of the present embodiment can also perform focus detection by a contrast detection method.

Therefore, even if the main mirror 130 and the sub mirror 131 are retracted out of the optical path as in the case of live view display or movie shooting, the camera 100 has a phase difference detection method and contrast based on the signal obtained from the image sensor 14. Focus detection of both detection methods is possible. Further, in the normal still image photographing in which the main mirror 130 and the sub mirror 131 are in the optical path, the camera 100 can perform the focus detection of the phase difference detection method by the focus detection device 105. As described above, the camera 100 can detect the focus in any of the states of still image shooting, live view display, and moving image shooting.

The memory control unit 22 controls the A / D converter 16, the timing generation circuit 18, the image processing unit 20, the image display memory 24, the D / A converter 26, the memory 30, and the compression / expansion unit 32. Then, the data of the A / D converter 16 is written to the image display memory 24 or the memory 30 via the image processing unit 20 and the memory control unit 22, or only through the memory control unit 22. The image data for display written in the image display memory 24 is displayed on the image display unit 28 composed of a liquid crystal monitor or the like via the D / A converter 26. The electronic viewfinder function (live view display) can be realized by sequentially displaying the moving images captured by the image sensor 14 on the image display unit 28. The image display unit 28 can turn on / off the display according to the instruction of the system control unit 50, and when the display is turned off, the power consumption of the camera 100 can be significantly reduced.

Further, the memory 30 is used for temporary storage of captured still images and moving images, and has a sufficient storage capacity for storing a predetermined number of still images and moving images for a predetermined time. As a result, even in the case of continuous shooting or panoramic shooting, a large amount of images can be written to the memory 30 at high speed. The memory 30 can also be used as a work area for the system control unit 50. The compression / decompression unit 32 has a function of compressing / decompressing image data by adaptive discrete cosine transform (ADCT) or the like, reads the image stored in the memory 30 and performs compression processing or decompression processing, and the processed image data. Is written back to the memory 30.

The shutter control unit 36 controls the shutter 12 based on the photometric information from the photometric unit 46 in cooperation with the aperture control unit 344 that controls the aperture 312 of the photographing lens 300. The interface unit 38 and the connector 122 electrically connect the camera 100 and the photographing lens 300. These have a function of transmitting a control signal, a state signal, a data signal, and the like between the camera 100 and the photographing lens 300, and also having a function of supplying currents of various voltages. Further, not only telecommunications but also optical communication, voice communication and the like may be transmitted.

The metering unit 46 performs automatic exposure control (AE) processing. The brightness of the subject optical image can be measured by causing the luminous flux that has passed through the photographing lens 300 to enter the photometric unit 46 via the lens mount 106, the main mirror 130, and a photometric lens (not shown). The photometric unit 46 can determine the exposure condition by using a program diagram or the like in which the subject brightness and the exposure condition are associated with each other. The photometric unit 46 also has a dimming processing function in cooperation with the flash 48. It is also possible for the system control unit 50 to perform AE control on the shutter control unit 36 and the aperture control unit 344 of the photographing lens 300 based on the calculation result obtained by calculating the image data of the image sensor 14 by the image processing unit 20. Is. The flash 48 also has an AF auxiliary light projection function and a flash dimming function.

The system control unit 50 has a programmable processor such as a CPU or MPU, and controls the operation of the entire camera system by executing a program stored in advance. The non-volatile memory 52 stores constants, variables, programs, and the like for the operation of the system control unit 50. The display unit 54 is, for example, a liquid crystal display device that displays an operating state, a message, or the like using characters, images, voices, or the like according to the execution of a program by the system control unit 50. A single or a plurality of display units 54 are installed at positions near the operation unit of the camera 100 so that they can be easily seen, and are composed of, for example, a combination of LCDs, LEDs, and the like. Among the display contents of the display unit 54, the information to be displayed on the LCD or the like includes information on the number of shots such as the number of recorded shots and the number of remaining shots, information on shooting conditions such as shutter speed, aperture value, exposure compensation, and flash. There is. In addition, the battery level, date, time, etc. are also displayed. Further, as described above, some of the functions of the display unit 54 are installed in the optical finder 104.

The non-volatile memory 56 is a memory that can be electrically erased and recorded, and for example, EEPROM or the like is used. Reference numerals 60, 62, 64, 66, 68 and 70 are operation units for inputting various operation instructions of the system control unit 50, such as a switch, a dial, a touch panel, pointing by line-of-sight detection, a voice recognition device, or the like. It consists of multiple combinations.

The mode dial 60 can switch and set each function mode such as power off, auto shooting mode, manual shooting mode, playback mode, and PC connection mode. The shutter switch SW1 62 turns on when the shutter button (not shown) is pressed halfway, and instructs the start of operations such as AF processing, AE processing, AWB processing, and EF processing. The shutter switch SW2 64 turns on when the shutter button is fully pressed, and instructs the start of a series of processing related to shooting. The series of processing related to photography includes exposure processing, development processing, recording processing, and the like. In the exposure process, the signal read from the image sensor 14 is written as image data in the memory 30 via the A / D converter 16 and the memory control unit 22. In the development process, development is performed using calculations in the image processing unit 20 and the memory control unit 22. In the recording process, image data is read from the memory 30, compressed by the compression / decompression unit 32, and written as image data on the recording medium 150 or 160.

The image display ON / OFF switch 66 can set ON / OFF of the image display unit 28. With this function, when taking a picture using the optical viewfinder 104, power saving can be achieved by cutting off the current supply to the image display unit 28 including the liquid crystal monitor and the like. The quick review ON / OFF switch 68 sets a quick review function that automatically reproduces the captured image data immediately after shooting. The operation unit 70 includes various buttons, a touch panel, and the like. Various buttons include a menu button, a flash setting button, a single shooting / continuous shooting / self-timer switching button, and an exposure compensation button.

The power supply control unit 80 includes a battery detection circuit, a DC / DC converter, a switch circuit for switching a block to be energized, and the like. It detects whether or not a battery is installed, the type of battery, and the remaining battery level, controls the DC / DC converter based on the detection result and the instruction of the system control unit 50, and applies the required voltage for the required period, including the recording medium. Supply to each part. The connectors 82 and 84 connect a power supply unit 86 including a primary battery such as an alkaline battery or a lithium battery, a NiCd battery, a NiMH battery, a secondary battery such as a lithium ion battery, an AC adapter, or the like to the camera 100.

The interfaces 90 and 94 have a function of connecting to a recording medium such as a memory card or a hard disk, and the connectors 92 and 96 make a physical connection to a recording medium such as a memory card or a hard disk. The recording medium attachment / detachment detection unit 98 detects whether or not a recording medium is attached to the connector 92 or 96. In the present embodiment, the interface and the connector to which the recording medium is attached are described as having two systems, but the interface and the connector may be configured to have any number of systems, one or more. Further, the configuration may include a combination of interfaces and connectors of different standards. Further, by connecting various communication cards such as a LAN card to the interface and the connector, the image data and the management information attached to the image data can be transferred to each other with other peripheral devices such as a computer and a printer.

The communication unit 110 has various communication functions such as wired communication and wireless communication. The connector 112 connects the camera 100 to another device by the communication unit 110, and is an antenna in the case of wireless communication. The recording media 150 and 160 are a memory card, a hard disk, and the like. The recording media 150 and 160 include recording units 152 and 162 composed of a semiconductor memory, a magnetic disk, and the like, interfaces 154 and 164 with the camera 100, and connectors 156 and 166 for connecting to the camera 100.

Next, the photographing lens 300 will be described. The photographing lens 300 is mechanically and electrically coupled to the camera 100 by engaging the lens mount 306 with the lens mount 106 of the camera 100. The electrical coupling is realized by the connector 122 and the connector 322 provided on the lens mount 106 and the lens mount 306. The lens 311 includes a focus lens for adjusting the focusing distance of the photographing lens 300. The focus control unit 342 adjusts the focus of the photographing lens 300 by driving the focus lens along the optical axis. The operation of the focus control unit 342 is controlled by the system control unit 50 through the lens system control unit 346. The aperture 312 adjusts the amount and angle of the subject light incident on the camera 100.

The connector 322 and the interface 338 electrically connect the photographing lens 300 to the connector 122 of the camera 100. The connector 322 also has a function of transmitting a control signal, a state signal, a data signal, and the like between the camera 100 and the photographing lens 300, and supplying currents of various voltages. The connector 322 may be configured to transmit not only telecommunications but also optical communication, voice communication, and the like.

The zoom control unit 340 drives the variable magnification lens of the lens 311 and adjusts the focal length (angle of view) of the photographing lens 300. If the photographing lens 300 is a single focus lens, the zoom control unit 340 does not exist. The aperture control unit 344 controls the aperture 312 based on the light measurement information from the light measuring unit 46 in cooperation with the shutter control unit 36 that controls the shutter 12.

The lens system control unit 346 has a programmable processor such as a CPU or MPU, and controls the operation of the entire photographing lens 300 by executing a program stored in advance. The lens system control unit 346 has a memory function for storing constants, variables, programs, and the like for the operation of the photographing lens. The non-volatile memory 348 stores identification information such as a number unique to the photographing lens, management information, functional information such as an open aperture value and a minimum aperture value, a focal length, and current and past setting values.

In the present embodiment, lens frame information corresponding to the state of the photographing lens 300 is also stored. The lens frame information is information on the radius of the frame opening that determines the luminous flux passing through the photographing lens, and information on the distance from the image sensor 14 to the frame opening. The aperture 312 is included in a frame that determines the light flux passing through the photographing lens, and the aperture of the lens frame component that holds the lens also corresponds to the frame. Further, since the frame for determining the luminous flux passing through the photographing lens differs depending on the focus position and zoom position of the lens 311, a plurality of lens frame information is prepared corresponding to the focus position and zoom position of the lens 311. Then, when the camera 100 performs focus detection using the focus detection device, the optimum lens frame information corresponding to the focus position and the zoom position of the lens 311 is selected and sent to the camera 100 through the connector 322.
The above is the configuration of the camera system of the present embodiment including the camera 100 and the photographing lens 300.

Next, the configuration of the image pickup device 14 will be described with reference to FIGS. 2 and 3.
FIG. 2A shows a circuit configuration example of a plurality of pixels included in the image pickup device 14 having a configuration capable of outputting a signal used for focus detection in the phase difference detection method. Here, a configuration will be described in which one pixel 200 is provided with two photodiodes PD201a and 201b as a plurality of photoelectric conversion regions or photoelectric conversion units sharing a microlens. However, more (eg, 4) photodiodes may be provided. The photodiode 201a (first photoelectric conversion unit) and the photodiode 201b (second photoelectric conversion unit) function as focus detection pixels and also as imaging pixels, as will be described later.

The transfer switches 202a and 202b, the reset switch 205, and the selection switch 206 may be composed of, for example, a MOS transistor. In the following description, these switches are N-type MOS transistors, but they may be P-type MOS transistors or other switching elements.

FIG. 2B is a diagram schematically showing horizontal n pixels and vertical m pixels among a plurality of pixels two-dimensionally arranged on the image sensor 14. Here, it is assumed that all the pixels have the configuration shown in FIG. 2 (a). A microlens 236 is provided for each pixel, and the photodiodes 201a and 201b share the same microlens. Hereinafter, the signal obtained by the photodiode 201a is referred to as an A signal or a first signal, and the signal obtained by the photodiode 201b is referred to as a B signal or a second signal. Further, the signal sequence for focus detection generated from a plurality of A signals is referred to as an A image or a first image signal, and the signal sequence for focus detection generated from a plurality of B signals is referred to as a B image or a second image signal. Call. Further, the paired A image and B image are referred to as a signal sequence pair or an image signal pair.

The transfer switch 202a is connected between the photodiode 201a and the floating diffusion (FD) 203. Further, the transfer switch 202b is connected between the photodiode 201b and the FD 203. The transfer switches 202a and 202b are elements that transfer the electric charges generated by the photodiodes 201a and 201b to the common FD203, respectively. The transfer switches 202a and 202b are controlled by the control signals TX_A and TX_B, respectively.

The floating diffusion (FD) 203 temporarily holds the charges transferred from the photodiodes 201a and 201b, and functions as a charge-voltage conversion unit (capacitor) that converts the held charges into a voltage signal.

The amplification unit 204 is a source follower MOS transistor. The gate of the amplification unit 204 is connected to the FD 203, and the drain of the amplification unit 204 is connected to the common power supply 208 that supplies the power supply potential VDD. The amplification unit 204 amplifies the voltage signal based on the electric charge held in the FD 203 and outputs it as an image signal.

The reset switch 205 is connected between the FD 203 and the common power supply 208. The reset switch 205 is controlled by the control signal RES and has a function of resetting the potential of the FD 203 to the power supply potential VDD.

The selection switch 206 is connected between the source of the amplification unit 204 and the vertical output line 207. The selection switch 206 is controlled by the control signal SEL and outputs the image signal amplified by the amplification unit 204 to the vertical output line 207.

FIG. 3 is a diagram showing a configuration example of the image sensor 14. The image sensor 14 includes a pixel array 234, a vertical scanning circuit 209, a current source load 210, a readout circuit 235, a common output line 228, 229, a horizontal scanning circuit 232, and a data output unit 233. In the following, it is assumed that all the pixels included in the pixel array 234 have the circuit configuration shown in FIG. 2A. However, some pixels may have a configuration in which one photodiode is provided per microlens.

The pixel array 234 has a plurality of pixels 200 arranged in a matrix. FIG. 3 shows a 4-row n-column pixel array 234 for the sake of brevity. However, the number of rows and columns of the pixels 200 included in the pixel array 234 is arbitrary. Further, in the present embodiment, the image pickup device 14 is a single-plate color image pickup device, and has a color filter having a primary color Bayer arrangement. Therefore, the pixel 200 is provided with any one of the red (R), green (G), and blue (B) color filters. There are no particular restrictions on the colors and arrangements that make up the color filter. Further, some pixels included in the pixel array 234 are shielded from light to form an optical black (OB) region.

The vertical scanning circuit 209 supplies various control signals shown in FIG. 2A to the pixels 200 of each row via the drive signal lines 236 provided for each row. Although the drive signal line 236 of each line is represented by one line in FIG. 3 for simplification, a plurality of drive signal lines actually exist in each line.

The pixels included in the pixel array 234 are connected to a common vertical output line 207 for each row. A current source load 210 is connected to each of the vertical output lines 207. The signal from each pixel 200 is input to the read-out circuit 235 provided for each column through the vertical output line 207.

The horizontal scanning circuit 232 outputs control signals hsr (0) to hsr (n-1), each of which corresponds to one reading circuit 235. The control signal hsr () selects one of the n readout circuits 235. The read-out circuit 235 selected by the control signal hsr () outputs a signal to the data output unit 233 through the common output lines 228 and 229.

Next, a specific circuit configuration example of the read circuit 235 will be described. FIG. 3 shows a circuit configuration example for one of the n read-out circuits 235, but the other read-out circuits 235 also have the same configuration. The readout circuit 235 of this embodiment includes a lamp-type AD converter.

The signal input to the readout circuit 235 through the vertical output line 207 is input to the inverting input terminal of the operational amplifier 213 via the clamp capacitance 211. A reference voltage Vref is supplied from the reference voltage source 212 to the non-inverting input terminal of the operational amplifier 213. Feedback capacitances 214 to 216 and switches 218 to 220 are connected between the inverting input terminal and the output terminal of the operational amplifier 213. A switch 217 is further connected between the inverting input terminal and the output terminal of the operational amplifier 213. The switch 217 is controlled by the control signal RES_C and has a function of short-circuiting both ends of the feedback capacitances 214 to 216. The switches 218 to 220 are controlled by the control signals GAIN0 to GAIN2 from the system control unit 50.

The output signal of the operational amplifier 213 and the lamp signal 224 output from the lamp signal generator 230 are input to the comparator 221. Latch_N222 is a storage element for holding a noise level (N signal), and Latch_S223 is a storage element for holding a signal level (A + B signal) obtained by adding an A signal and an A signal and a B signal. The output of the comparator 221 (value representing the comparison result) and the output of the counter 231 (counter value) 225 are input to each of Latch_N222 and Latch_S223, respectively. The operation (valid or invalid) of Latch_N222 and Latch_S223 is controlled by LATEN_N and LATEN_S, respectively. The noise level held by Latch_N222 is output to the common output line 228 via the switch 226. The signal level held by Latch_S223 is output to the common output line 229 via the switch 227. The common output lines 228 and 229 are connected to the data output unit 233.

The switches 226 and 227 are controlled by the control signal hsr (h) from the horizontal scanning circuit 232. Here, h indicates the column number of the read circuit 235 to which the control signal line is connected. The signal levels held in Latch_N222 and Latch_S223 of each read circuit 235 are sequentially output to the common output lines 228 and 229, and are output to the memory control unit 22 and the image processing unit 20 through the data output unit 233. This operation of sequentially outputting the signal levels held by each read-out circuit 235 to the outside is called horizontal transfer. The control signals (excluding hsr ()) input to the readout circuit and the control signals of the vertical scanning circuit 209, the horizontal scanning circuit 232, the lamp signal generator 230, and the counter 231 are the timing generation circuit 18 and the system control unit. Supplied from 50.

The reading operation for one row of pixels will be described with reference to FIG. 4, which is a timing chart relating to the reading operation of the image pickup device 14 shown in FIG. It is assumed that each switch is turned on when each control signal is H, and each switch is turned off when each control signal is L.

At time t1, the vertical scanning circuit 209 changes the control signals TX_A and TX_B from L to H with the control signal RES set to H, and turns on the transfer switches 202a and 202b. As a result, the electric charge accumulated in the photodiodes 201a and 201b is transferred to the common power supply 208 via the transfer switches 202a and 202b and the reset switch 205, and the photodiodes 201a and 201b are reset. The FD 203 is also reset in the same manner. When the vertical scanning circuit 209 sets the control signals TX_A and TX_B to L and turns off the transfer switches 202a and 202b at time t2, the storage of light charges starts at the photodiodes 201a and 201b.

When the predetermined accumulation time elapses, the vertical scanning circuit 209 sets the control signal SEL to H and turns on the selection switch 206 at time t3. As a result, the source of the amplification unit 204 is connected to the vertical output line 207. At time t4, the vertical scanning circuit 209 sets the control signal RES to L and turns off the reset switch 205. As a result, the reset of the FD 203 is released, and the reset signal level of the FD 203 is read out to the vertical output line 207 via the amplification unit 204 and input to the read circuit 235.

After that, at time t5, the timing generation circuit 18 sets the control signal RES_C to L. As a result, the switch 217 is turned on, and the voltage based on the difference between the reset signal level read out on the vertical output line 207 and the reference voltage Vref is output from the operational amplifier 213. Based on the ISO sensitivity set in advance by the operation unit 70, the image sensor 14 is set so that the system control unit 50 sets any one of the control signals GAIN0 to GAIN2 to H. For example, when the camera 100 of the embodiment can be set to any of ISO sensitivity 100, 200, and 400, the control signals GAIN0 are H, GAIN1, and GAIN2 are L when the ISO sensitivity is 100. Similarly, at ISO sensitivity 200, the control signal GAIN1 is H, and at ISO sensitivity 400, control signal GAIN2 is H. The type of set sensitivity and the relationship between the set sensitivity and the control signal are not limited to this.

The operational amplifier 213 amplifies the input voltage with an inverting gain determined by the capacitance ratio of the clamp capacitance 211 and one of the feedback capacitances 214 to 216 corresponding to the switch corresponding to the H of the control signals GAIN0 to GAIN2. And output. By this amplification, the random noise component generated in the circuit up to the operational amplifier 213 is also amplified. Therefore, the magnitude of random noise contained in the amplified signal depends on the ISO sensitivity.

Next, at time t6, the lamp signal generator 230 starts outputting a lamp signal whose signal level linearly increases with the passage of time, and at the same time, the counter 231 starts counting up from the reset state. Further, the timing generation circuit 18 sets LATEN_N to H and enables Latch_N. The comparator 221 compares the output signal of the operational amplifier 213 with the lamp signal output by the lamp signal generator 230. When the lamp signal level exceeds the output signal level of the operational amplifier 213, the output of the comparator 221 changes from L to H (time t7). When the output of the comparator 221 changes from L to H in the state where LATEN_N is H, the Latch_N222 stores the counter value output by the counter 231 at that time. The counter value stored in Latch_N222 corresponds to a digital value (N signal data) representing the N signal level. Since LATEN_S is L, Latch_S223 is invalid and the count value is not stored. After that, at time t8, when the lamp signal level reaches a predetermined value, the lamp signal generator 230 stops the output of the lamp signal, and the timing generator sets LATEN_N to L.

At time t9, the vertical scanning circuit 209 sets the control signal TX_A to H. As a result, the transfer switch 202a is turned on, and the optical charge (A signal) accumulated in the photodiode 201a is transferred to the FD 203 from time t2. After that, at time t10, the vertical scanning circuit 209 sets the control signal TX_A to L. The FD 203 converts the transferred charge into an electric potential, and this electric potential (A signal level) is output to the readout circuit 235 via the amplification unit 204 and the vertical output line 207. The operational amplifier 213 outputs a voltage based on the difference between the A signal level read out on the vertical output line 207 and the reference voltage Vref. The inverting gain of the operational amplifier 213 is determined by the ratio of the clamp capacitance 211 to any one of the feedback capacitances 214 to 216.

Next, at time t11, the lamp signal generator 230 starts outputting the lamp signal, and at the same time, the counter 231 starts counting up from the reset state. Further, the timing generation circuit 18 sets LATEN_S to H and enables Latch_S. The comparator 221 compares the output signal of the operational amplifier 213 with the lamp signal output by the lamp signal generator 230. When the lamp signal level exceeds the output signal level of the operational amplifier 213, the output of the comparator 221 changes from L to H (time t12). When the output of the comparator 221 changes from L to H in the state where LATEN_S is H, the Latch_S223 stores the counter value output by the counter 231 at that time. The counter value stored in Latch_S223 corresponds to a digital value (A signal data) representing the A signal level. Since LATEN_N is L, Latch_N222 is invalid and the count value is not stored. After that, at time t13, when the lamp signal level reaches a predetermined value, the lamp signal generator 230 stops the output of the lamp signal, and the timing generator sets LATEN_S to L.

After that, during the time t14 to t15, the horizontal scanning circuit 232 sequentially sets the control signal hsr (h) to H for a certain period of time. As a result, the switches 226 and 227 of each read circuit 235 are turned on for a certain period of time and then turned off. The N signal data and the A signal data held in Latch_N222 and Latch_S223 of each read circuit 235 are read out to the common output lines 228 and 229, respectively, and input to the data output unit 233. The data output unit 233 outputs a value obtained by subtracting the N signal data from the A signal data to the outside with respect to the A signal data and the N signal data output from each read circuit 235.

From time t16 to t17, the vertical scanning circuit 209 sets the control signals TX_A and TX_B to H and turns on the transfer switches 202a and 202b. As a result, the optical charge is transferred from both photodiodes 201a and 201b to the FD203. The FD 203 converts the transferred charge into an electric potential, and this electric potential (A + B signal level) is output to the readout circuit 235 via the amplification unit 204 and the vertical output line 207. The operational amplifier 213 outputs a voltage based on the difference between the A + B signal level read out on the vertical output line 207 and the reference voltage Vref.

Next, at time t18, the lamp signal generator 230 starts outputting the lamp signal, and at the same time, the counter 231 starts counting up from the reset state. Further, the timing generation circuit 18 sets LATEN_S to H and enables Latch_S. The comparator 221 compares the output signal of the operational amplifier 213 with the lamp signal output by the lamp signal generator 230. When the lamp signal level exceeds the output signal level of the operational amplifier 213, the output of the comparator 221 changes from L to H (time t19). When the output of the comparator 221 changes from L to H in the state where LATEN_S is H, the Latch_S223 stores the counter value output by the counter 231 at that time. The counter value stored in Latch_S223 corresponds to a digital value (A + B signal data) representing the A + B signal level. After that, at time t20, when the lamp signal level reaches a predetermined value, the lamp signal generator 230 stops the output of the lamp signal, and the timing generator sets LATEN_S to L.

After that, during the time t21 to t22, the horizontal scanning circuit 232 sequentially sets the control signal hsr (h) to H for a certain period of time. As a result, the switches 226 and 227 of each read circuit 235 are turned on for a certain period of time and then turned off. The N signal data and the A + B signal data held in Latch_N222 and Latch_S223 of each read circuit 235 are read out to the common output lines 228 and 229, respectively, and input to the data output unit 233. The data output unit 233 outputs a value obtained by subtracting the N signal data from the A + B signal data to the outside with respect to the A + B signal data and the N signal data output from each read circuit 235.

When the timing generation circuit 18 sets the control signal RES_C to H at time t22, the vertical scanning circuit 209 sets the control signal RES to H at time t23, and the vertical scanning circuit 209 sets the control signal SEL to L at time t24, one line is equivalent. The read operation is completed. By repeating this operation for a predetermined number of lines, an image signal for one screen is acquired.

The camera 100 of the present embodiment has a still image mode and a moving image mode. When the still image mode is set, the system control unit 50 controls to read pixel data for all lines from the image sensor 14. Further, if the moving image mode is set, the system control unit 50 controls to read pixel data from the image sensor 14 in a cycle of, for example, three lines (reading one line and skipping two lines). As described above, in the present embodiment, the number of lines to be read is smaller in the moving image mode than in the still image mode. However, the reading method in the still image mode and the moving image mode is not limited to this.

In this way, the A signal and the A + B signal from which the reset noise has been removed can be read out from the image sensor 14. The A signal is used as a signal for focusing detection, and the A + B signal is used as a signal constituting a captured image. The A + B signal and the A signal are also used to generate a B signal for focus detection.

The image sensor 14 of the present embodiment has two types of read modes, an all-pixel read mode and a thinned read mode. The all-pixel reading mode is a mode for reading all effective pixels, and is set when, for example, a high-definition still image is obtained.

The thinning-out reading mode is a mode for reading out fewer pixels than the all-pixel reading mode. For example, when obtaining pixels having a lower resolution than a high-definition still image such as a moving image or an image for preview, it is necessary to read out at high speed. Set in case. For example, pixels can be thinned out at the same ratio in both the horizontal and vertical directions so as not to change the aspect ratio of the image. The "thinning out" includes not only a configuration in which the reading itself is not performed, but also a configuration in which the read signal is discarded (ignored) and a configuration in which one signal is generated from a plurality of read signals. For example, the S / N can be improved by averaging the signals read from a plurality of adjacent pixels to generate one signal.

FIG. 5A shows photoelectric conversion units 201a and 201b of pixels 200 (center pixels) arranged near the center of the exit pupil surface of the photographing lens 300 and the image plane of the image sensor 14 in the image pickup apparatus of the present embodiment. It is a figure explaining the conjugate relationship with. The photoelectric conversion units 201a and 201b in the image sensor 14 and the exit pupil surface of the photographing lens 300 are designed to be in a conjugated relationship by the on-chip microlens 201i. The exit pupil surface of the photographing lens 300 generally coincides with the surface on which the iris diaphragm for adjusting the amount of light is provided.

On the other hand, the photographing lens 300 of the present embodiment is a zoom lens having a variable magnification function. Some zoom lenses change the size of the exit pupil and the distance from the image plane to the exit pupil (exit pupil distance) when the magnification operation is performed. FIG. 5 shows a state in which the focal length of the photographing lens 300 is at the center of the wide-angle end and the telephoto end. With the exit pupil distance Zep in this state as a standard value, the optimum design of the eccentricity parameter is made according to the shape of the on-chip microlens and the image height (distance from the center of the screen or XY coordinates).

In FIG. 5A, the photographing lens 300 includes a first lens group 101, a lens barrel member 101b that holds the first lens group, a third lens group 105, and a lens barrel member 105b that holds the third lens group. is doing. Further, the photographing lens 300 has an aperture 102, an aperture plate 102a that defines the aperture diameter when the aperture is open, and an aperture blade 102b for adjusting the aperture diameter when the aperture is stopped down. In FIG. 5, 101b, 102a, 102b, and 105b, which act as limiting members for the light flux passing through the photographing lens 300, show an optical virtual image when observed from the image plane. Further, the composite aperture in the vicinity of the aperture 102 is defined as the exit pupil of the photographing lens 300, and the distance from the image plane is defined as the exit pupil distance Zep.

Photoelectric conversion units 201a and 201b are arranged on the bottom layer of the pixel 200. Wiring layers 201e to 201g, a color filter 201h, and an on-chip microlens 201i are provided on the upper layers of the photoelectric conversion units 201a and 201b. The photoelectric conversion units 201a and 201b are projected onto the exit pupil surface of the photographing lens 300 by the on-chip microlens 201i. In other words, the exit pupil is projected onto the surfaces of the photoelectric conversion units 201a and 201b via the on-chip microlens 201i.

FIG. 5B shows the projected images EP1a and EP1b of the photoelectric conversion units 201a and 201b on the exit pupil surface of the photographing lens 300. The circular TL indicates the maximum incident range of the luminous flux on the pixel 200, which is defined by the opening plate 102a of the aperture 102, on the exit pupil surface. Since the circle TL is defined by the opening plate 102a, the circle TL is also referred to as 102a in the figure. Since FIG. 5 shows the central pixel, the eclipse of the luminous flux is symmetrical with respect to the optical axis, and the photoelectric conversion units 201a and 201b receive the luminous flux that has passed through the pupil region of the same size. Since the circular TL contains most of the projected images EP1a and EP1b, eclipse of the luminous flux hardly occurs. Therefore, when the photoelectrically converted signals are added by the photoelectric conversion units 201a and 201b, the result of photoelectric conversion of the circular TL, that is, the luminous flux that has passed through almost the entire exit pupil region can be obtained. The region of the exit pupil received by the photoelectric conversion unit 201a is the first pupil region, the region of the exit pupil received by the photoelectric conversion unit 201b is the second pupil region, and the first pupil region and the second pupil region are combined. The region is called the third pupil region.

As described above, the image sensor 14 of the present embodiment has a function of a focus detection sensor for phase difference AF in addition to a function of acquiring a captured image. As described above, since the signals obtained by the plurality of photoelectric conversion units can be used as the output of a normal image pickup pixel by collecting the signals for each pixel, the contrast AF is performed using the output of the image pickup element 14 (image pickup image signal). You can also do it.

FIG. 6 is a diagram showing a focus detection region 401 within the photographing range 400, and the phase difference detection method AF is performed by the image pickup device 14 within this focus detection region (focus detection sensor on the image pickup surface (light receiving surface)). ..

FIG. 6 is a diagram showing an example of a focus detection area 401 set in the shooting range 400. When focus detection is performed using the output of the pixels of the image sensor 14, the output of the pixels included in the area of the image sensor 14 corresponding to the focus detection area 401 is output in both the contrast detection method and the phase difference detection method. Use. Therefore, it can be said that the focus detection region 401 is set in the image sensor 14, and the focus detection region 401 will be described below as a pixel region of the image sensor 14 for ease of explanation and understanding. Further, in the focus detection area 401, it is assumed that the pixels 200 having the configuration shown in FIG. 2A are arranged in 1 row and 4N columns. Note that this is merely an example, and the number and size (number of pixels included) of the focus detection region can be appropriately determined within a range that does not interfere with the phase difference detection.

FIG. 7 schematically shows the signals output from the 1-row, 4N-column pixels 200 and the pixels 200 arranged in the focus detection area 401. In the following, the pixels (photodiode 201a) used to create the signal of the AF image A in the i-th row and j-th column and their outputs are referred to as A (i, j). Similarly, the pixel (photodiode 201b) used to create the signal of the AF B image in the i-th row and j-th column and its output are represented by B (i, j). Since the number of lines is 1 in FIG. 7, the notation of i is omitted. Therefore, A (1, j) and B (1, j) in the following description correspond to A (j) B (j) in FIG. 7.

In the focus detection of the phase difference detection method, a pair of images having a portion corresponding to the same subject is generated, the phase difference of the pair of images is detected, and the phase difference is converted into a defocus amount and a direction. A signal sequence (A image) based on the A signal obtained from the photodiode 201a of a plurality of pixels 200 existing in a predetermined direction (for example, the horizontal direction) and a signal sequence (B image) based on the B signal obtained from the photodiode 201b. Corresponds to an image of the same subject viewed from different viewpoints. Therefore, by detecting the phase difference between the A image and the B image and converting it into the defocus amount and the direction, the focus detection of the phase difference detection method can be realized.

Then, while changing the relative distance (shift amount) between the A image and the B image in the predetermined direction described above, a value (correlation amount) representing the correlation between the A image and the B image at each position is calculated, and the correlation is calculated. The highest shift amount can be detected as the phase difference between the A image and the B image. The correlation amount may be, for example, the difference cumulative value of the corresponding signal values, but may be another value.

そして、ＣＯＲ（ｋ）を最小にするシフト量ｋの値を求める。ここで、式（１）で算出するシフト量ｋは整数であるが、分解能を向上させるため、最終的に求めるのシフト量ｋは実数とする。例えば式（１）で得られる最小値がＣＯＲ（ａ）の場合、ＣＯＲ（ａ−１）、ＣＯＲ（ａ）、ＣＯＲ（ａ＋１）からの補間演算などにより、この区間での相関量を最小にする実数値のシフト量を求める。 For example, in the example of FIG. 7, an A image is generated from A (1) to A (4N), and a B image is generated from B (1) to B (4N), and the shift amount k is set to −kmax ≦ k ≦ in pixel units. Assuming that the value is changed within the range of kmax, the correlation amount COR (k) at each relative position can be calculated as follows.

Then, the value of the shift amount k that minimizes the COR (k) is obtained. Here, the shift amount k calculated by the equation (1) is an integer, but in order to improve the resolution, the shift amount k finally obtained is a real number. For example, when the minimum value obtained by the equation (1) is COR (a), the amount of correlation in this interval is minimized by interpolation operations from COR (a-1), COR (a), and COR (a + 1). Find the amount of shift of the real value to be performed.

Here, a problem that may occur in the amount of correlation obtained when the B signal is generated from the A + B signal and the A signal will be described. Here, the A image generated from A (1) to A (4N) is referred to as S [A], and the A + B image generated from A + B (1) to A + B (4N) is referred to as S [A + B]. Further, the random noise superimposed when A (1) to A (4N) is read is referred to as N [A], and the random noise superimposed when A + B (1) to A + B (4N) is read is referred to as N [A + B]. show. Random noise is due to the readout circuit.

The B image generated from the A + B image and the A image can be expressed by the following equation (2).
B image = A + B image-A image = (S [A + B] + N [A + B])-(S [A] + N [A])
= (S [A + B] -S [A]) + (N [A + B] -N [A])
… (2)
Will be.

At this time, the correlation amount COR (s) when the shift amount k = s (s ≠ 0) can be expressed by the following equation (3).
COR (s) = Σ | A (is) -B (i + s) |
= Σ | {S [A (is)] + N [A (is)]}-{S [A + B (i + s)]
-S [A (i + s)] + N [A + B (i + s)]-N [A (i + s)]} |
= Σ | S [A (i-s)] + S [A (i + s)]-S [A + B (i + s)]
+ N [A (is)] + N [A (i + s)]-N [A + B (i + s)] |
… (3)

On the other hand, the correlation amount COR (0) when the shift amount k = 0 can be expressed by the following equation (4).
COR (0) = Σ | A (i) -B (i) |
= Σ | S [A (i)] + S [A (i)]-S [A + B (i)]
+ N [A (i)] + N [A (i)]-N [A + B (i)] |
= Σ | 2 × S [A (i)] −S [A + B (i)]
+ 2 × N [A (i)]-N [A + B (i)] |
… (4)

Here, in the case where the shift amount k = s (≠ 0) and the case where the shift amount k = 0, the random noise components Noise (s) and Noise (0) included in the correlation amount COR are expressed by the following equations, respectively. It can be represented by (5) and (6).
-When the shift amount k = s (≠ 0) Noise (s) = Σ | N [A (i−s)] + N [A (i + s)] −N [A + B (i + s)] |… (5)
・ When the shift amount k = 0 Noise (0) = Σ | 2 × N [A (i)] −N [A + B (i)] |

Here, N [A (is)], N [A (i + s)], and N [A + B (i + s)] constituting Noise (s) are random noises that are not correlated with each other. Therefore, Noise (s) becomes a substantially constant value. On the other hand, N [A (i)] and N [A + B (i)] constituting Noise (0) are random noises that do not correlate with each other, but N [A (i)] is doubled. , Noise (0) is larger than Noise (s). Therefore, when the relationship between the shift amount k and the noise (k) is schematically represented, the noise component in the correlation amount becomes large only when the shift amount k = 0 as shown in FIG. 11A.

The random noise superimposed on the A image, which has the same absolute value as the random noise superimposed on the A image, is superimposed on the B image generated by subtracting the A image from the A + B image. As described above, the B image generated by subtracting the A image from the A + B image contains random noise having a correlation with the random noise superimposed on the A image. Therefore, the correlation between the random noise of the A image and the B image is specifically increased when the shift amount k = 0. In the calculation of the correlation amount COR (0) when the shift amount k = 0, the noise components of | 2 × N [A (i)] | included in the differences between the individual signals are integrated, and Σ | 2 × N A peak corresponding to [A (i)] | is generated.

When the contrast of the subject is low or the environmental brightness is low, the SN ratio of the obtained image is lowered, so that the noise component contained in the correlation amount COR of the A image and the B image is increased. For example, assume a state of focusing with a shift amount k = 0. In this case, if there is no influence of noise, the value of the correlation amount COR (k) with respect to the shift amount k changes as shown in FIG. 11B, and the correlation amount COR (k) becomes the minimum (the highest correlation). The shift amount k = 0 at the point H can be detected correctly. On the other hand, when the noise component has a peak at the shift amount k = 0 as shown in FIG. 11 (a), the value of the correlation amount COR (k) with respect to the shift amount k changes as shown in FIG. 11 (c). In this case, since the correlation amount COR (k) becomes the maximum (point I) when the shift amount k = 0, the shift amount to be detected cannot be detected correctly. Furthermore, since COR (k) is minimized at two locations near the shift amount k = 0 (points J and K), these points are recognized as the shift amount at the in-focus position, resulting in false detection or hunting (focus). The problem arises that the lens repeatedly moves back and forth).

The amount of noise contained in the B image generated by subtracting the A image from the A + B image had a negative correlation with the amount of noise contained in the A image. However, a problem arises even when the amount of noise contained in the A image and the B image has a positive correlation. For example, there are variations in the sensitivity of each pixel due to the transmittance of the color filter of the pixels and the variations in the characteristics of the readout circuit. Therefore, the A image and the A + B image read from the same pixel share a noise source. In this case, the larger the signal amount, the larger the noise amount. Therefore, there is a difference in the noise amount between the A image and the A + B image according to the difference in the signal amount.

The A image is represented by S [A], and the noise component having different values between pixels is represented by N [A]. In this case, it is assumed that only a noise component having a positive correlation with the signal amount is generated in the A image and the A + B image. When the signal amount S [A + B] of the A + B image is expressed as g × S [A] (g ≧ 1) as a relative amount with respect to the signal amount S [A] of the A image, the noise amount N [A + B] of the A + B image is It can be expressed as g × N [A] as a relative amount with respect to the noise amount N [A] of the A image.

In this case, the B image obtained by subtracting the A image from the A + B image can be represented by the following equation (7).
Image B = (S [A + B] + N [A + B])-(S [A] + N [A])
= (G-1) (S [A] + N [A]) ... (7)

When the correlation amount COR (k) between the A image and the B image is calculated according to the equation (1), the noise amount Noise (s) when the shift amount k = s (≠ 0) and the noise when the shift amount k = 0 are calculated. The quantity Noise (0) can be represented by the following equations (8) and (9).
-When the shift amount k = s (≠ 0) Noise (s) = Σ | N [A (i-s)]-(g-1) x N [A (i + s)] |
・ When the shift amount k = 0 Noise (0) = Σ | (2-g) × N [A (i)] |… (9)

Here, N [A (is)] and N [A (i + s)] constituting Noise (s) are random noises having the same variation as N [A] and having no correlation with each other. Therefore, N [A (is)]-(g-1) x N [A (i + s)] has a larger variation than N [A (l)]. Compared to the Noise (s) obtained by integrating the variation, the Noise (0) obtained by integrating the variation of N [A] becomes smaller frequently.

In the focus detection of the phase difference detection method, the defocus state is detected by detecting the shift amount k that minimizes the correlation amount COR (k) (maximizes the correlation). Therefore, if the correlation amount of the noise component is smaller than the other shift amounts at the shift amount k = 0, the shift amount k = 0 may be erroneously detected as the shift amount that minimizes the correlation value COR (k). .. When the difference in the amount of signals between the A image and the A + B image is large, this problem is likely to occur, for example, in an image of a subject that is uniformly bright with low contrast. Further, when the color and brightness of the subject are uniform, the focus detection by the phase difference detection method is originally impossible, but the defocus amount and direction based on the shift amount that minimizes the correlation amount of the noise component are determined. It may be detected.

Also in this embodiment, the A (or B) signal is generated from the A + B signal and the B (or A) signal read from the same pixel. Therefore, the noise components of the A signal and the B signal corresponding to the same pixel have a negative or positive correlation. However, in order to suppress the decrease in focus detection accuracy due to the correlation of the noise component between the A image and the B image as described above becoming a specific value at a specific shift amount, a plurality of types of A signal and B signal are used. A image and B image are generated and used for correlation calculation.

Specifically, in the present embodiment, the correlation amount is calculated for a plurality of A image B image pairs (As_1 and Bs_1, As_1 and Bs_2, As_3 and Bs_3). For the purpose of reducing the calculation load, improving the S / N of the output signal, adjusting the output image size, etc., the signal sequence is generated from the added pixel signal obtained by adding the outputs of a plurality of pixels (here, 2 pixels). do.

Here, the individual signals constituting the first A image As_1 generated from the pixels in the i-th row are As_1 (i, k), and the individual signals constituting the first B image Bs_1 are Bs_1 (i, k). ) (K is the number of the signal constituting each signal sequence). In this case, As_1 (1, k) and Bs_1 (1, k) for the pixels of 1 row and 4 N columns shown in FIG. 7 can be expressed by the following equation (10).
As_1 (1, k) = A (1,2 × (k-1) +1) + A (1,2 × (k-1) +2)
Bs_1 (1, k) = B (1,2 × (k-1) +1) + B (1,2 × (k-1) +2)
(1 ≦ k ≦ 2N, k is an integer)… (10)

As described above, the first A image As_1 and the first B image Bs_1 have the additional output of the first pixel and the second pixel, the additional output of the third pixel and the fourth pixel, ... As described above, it is composed of 2N signals obtained by adding the outputs of two pixels of the same type adjacent to each other in the horizontal direction.

Further, the individual signals constituting the second A image As_2 generated from the pixels in the i-th row are As_2 (i, m), and the individual signals constituting the second B image Bs_2 are Bs_2 (i, m). (M is the number of the signal constituting each signal sequence). In this case, As_2 (1, m) and Bs_2 (1, m) for the pixels of 1 row and 4 N columns shown in FIG. 7 can be expressed by the following equation (11).
As_1 (1, m) = As_1 (1,2 m-1)
Bs_1 (1, m) = Bs_1 (1,2 m)
(1 ≦ m ≦ N)… (11)

Further, the individual signals constituting the third A image As_3 generated from the pixels in the i-th row are As_3 (i, m), and the individual signals constituting the third B image Bs_3 are Bs_3 (i, m). (M is the number of the signal constituting each signal sequence). In this case, As_3 (1, m) and Bs_3 (1, m) for the pixels of 1 row and 4 N columns shown in FIG. 7 can be expressed by the following equation (12).
As_3 (1, m) = As_1 (1,2 m)
Bs_3 (1, m) = Bs_1 (1,2 m-1)
(1 ≦ m ≦ N)… (12)

As described above, the second A image As_2 is composed of the odd-numbered signals constituting the first A image As_1, and the second B image Bs_2 is composed of the even-numbered signals constituting the first B image Bs_1. Will be done. Further, the third A image As_3 is composed of even-numbered signals constituting the first A image As_1, and the third B image Bs_3 is composed of odd-numbered signals constituting the first B image Bs_1. .. In other words, between the second A image As_2 and the B image Bs_2, and between the third A image As_3 and the B image Bs_3, the sampling position of the subject is set by half the pitch of the sampling pitch in the phase difference detection direction. It is out of alignment.

That is, the pixel group used for generating the second A image As_2 and the pixel group used for generating the second B image Bs_2 are different from each other. Therefore, the correlation between the noise component of the second A image As_2 and the noise component of the second B image Bs_2 is low. The same applies to the third A image As_3 and the third B image Bs_3. Therefore, the A image and B are calculated by calculating the correlation amount COR (k) using the second A image As_2 and the second B image Bs_2, and the third A image As_3 and the third B image Bs_3. It is possible to suppress the problem caused by the correlation with the noise component of the image showing a specific value at a specific shift amount.

As described above, in the present embodiment, the pair of signal sequences (first A image As_1 and first B image Bs_1) that are correlated with each other's noise components based on the output signal of the first pixel group. Calculate the amount of correlation. Further, the correlation amount is calculated for a pair of signal sequences (second A image As_2 and second B image Bs_2) based on the output signals of the second pixel group and the third pixel group constituting the first pixel group. calculate. Further, the correlation amount is calculated for another pair of signal sequences (third B image Bs_3 and third A image As_3) based on the output signals of the second pixel group and the third pixel group. Here, as an example in which the correlation of noise components is the lowest, it is assumed that the second pixel group and the third pixel group do not overlap, but some overlap is not excluded.

(Focus detection operation)
Next, the focus adjustment operation in the camera 100 will be described with reference to the flowchart shown in FIG. The process shown in FIG. 8 is a state in which the main mirror 130 and the sub mirror 131 are retracted (mirror up) out of the optical path, and more specifically, during live view display (when shooting a moving image for display) or when recording a moving image (for recording). This is the process that is performed when shooting a moving image. Although the phase difference detection method of automatic focus detection using the output of the image sensor 14 is described here, the contrast detection method of automatic focus detection can also be performed as described above.

In S501, the system control unit 50 determines whether the focus detection start instruction has been input by the operation of SW162, the operation unit 70, etc., and if it is determined that the focus detection start instruction has been input, advances the process to S502 and inputs the input. If it is not determined, wait. The system control unit 50 may proceed to S502 not only by inputting the focus detection start instruction but also by using the start of the live view display or the moving image recording as a trigger.

In S502, the system control unit 50 acquires various lens information such as the lens frame information of the photographing lens 300 and the focus lens position from the lens system control unit 346 via the interface units 38, 338 and the connectors 122, 322.
In S503, the system control unit 50 generates image signal pairs (first to third A images and B images) for AF from the pixel data in the focus detection region of the frame image data read sequentially. The image processing unit 20 is instructed to do so. The image processing unit 20 generates an image signal pair for AF and supplies it to the AF unit 42. The AF unit 42 performs processing for correcting the difference in signal level with respect to the image signal pair for AF. Further, the AF unit 42 detects the peak value (maximum value) and the bottom value (minimum value) of the image signal for AF.

In S504, the AF unit 42 calculates, for example, the above-mentioned correlation amount COR (k) for each of the first A image and B image, the second A image and B image, and the third A image and B image, and correlates them. The shift amount k that minimizes the value COR (k) is detected as the phase difference of the image. Then, the AF unit 42 converts the detected phase difference into a defocus amount. The details of this process will be described later. The AF unit 42 outputs the defocus amount to the system control unit 50.
The system control unit 50 as an adjustment means in S505 determines the focus lens drive amount and drive direction of the photographing lens 300 based on the defocus amount obtained from the AF unit 42 in S504.

In S506, the system control unit 50 transmits information on the focus lens drive amount and drive direction to the lens system control unit 346 of the photographing lens 300 via the interface unit 38, 338, the connector 122, and 322. The lens system control unit 346 transmits information on the focus lens drive amount and drive direction to the focus control unit 342. The focus control unit 342 drives the focus lens based on the received information on the lens drive amount and the drive direction. As a result, the focus of the photographing lens 300 is adjusted. The operation of FIG. 8 may be continuously performed even when the moving image data of the next frame transition is read. Information on the focus lens drive amount and the drive direction may be directly transmitted from the system control unit 50 to the focus control unit 342.

Next, the calculation process of the defocus amount performed by the AF unit 42 in S504 of FIG. 8 will be further described with reference to the flowchart shown in FIG. In S5041, the AF unit 42 calculates the correlation amount COR1 (k) with respect to the first A image As_1 and the first B image Bs_1 in the same manner as in the equation (1). After the AF unit 42 obtains the correlation amount COR1 (k) for each shift amount k, the shift amount k at which the correlation between the first A image As_1 and the first B image Bs_1 is highest, that is, the correlation amount COR1 Find the value of the shift amount k that minimizes (k). The shift amount k at the time of calculating the correlation amount COR1 (k) is an integer, but when the shift amount k that minimizes the correlation amount COR1 (k) is obtained, the accuracy of the defocus amount is improved. Interpolation processing is performed as appropriate to obtain a value (real value) in sub-pixel units.

In the present embodiment, the shift amount dk in which the sign of the difference value of the correlation amount COR1 changes is calculated as the shift amount k that minimizes the correlation amount COR1 (k).
First, the AF unit 42 calculates the difference value DCOR1 of the correlation amount according to the following equation (13).
DCOR1 (k) = COR1 (k) -COR1 (k-1) ... (13)
Then, the AF unit 42 uses the difference value DCOR1 of the correlation amount to obtain the shift amount dk1 in which the sign of the difference amount changes. Assuming that the value of k immediately before the sign of the difference amount changes is k1 and the value of k whose sign changes is k2 (k2 = k1 + 1), the AF unit 42 calculates the shift amount dk1 according to the following equation (14). ..
dk1 = k1 + | DCOR1 (k1) | / | DCOR1 (k1) -DCOR1 (k2) | ... (14)

As described above, the AF unit 42 calculates the shift amount dk1 that maximizes the correlation amount between the first A image As_1 and the first B image Bs_1 in sub-pixel units, and finishes the process of S5041. The method for calculating the phase difference between the two one-dimensional image signals is not limited to the one described here, and any known method can be used.

In S5042, the AF unit 42 calculates the difference DCOR2 of the correlation amount COR2 (k) between the second A image As_2 and the second B image Bs_2 in the same manner as in S5041, and calculates the difference DCOR2 between the second A image As_2 and the second B image Bs_2. The shift amount dk2 that maximizes the correlation amount of the B image Bs_2 is calculated in sub-pixel units.

In S5043, the AF unit 42 calculates the difference DCOR3 of the correlation amount COR3 (k) between the third A image As_3 and the third B image Bs_3 in the same manner as in S5041, and calculates the difference DCOR3 between the third A image As_3 and the third B image Bs_3. The shift amount dk3 that maximizes the correlation amount of the B image Bs_3 is calculated in sub-pixel units.

In S5044, the AF unit 42 multiplies each of the shift amounts dk1, dk2, and dk3 calculated in S5041 to S5043 by a predetermined defocus conversion coefficient to convert them into defocus amounts Def1, Def2, and Def3. Instead of multiplying the conversion coefficient each time, the defocus amount may be obtained by using a table or the like in which the shift amount and the defocus amount after conversion are related. Here, the defocus conversion coefficient is obtained from the optical conditions at the time of shooting (aperture, exit pupil distance, lens frame information, etc.), the image height of the focus detection region, the sampling pitch of the signals constituting the A image and the B image, and the like. be able to. In the present embodiment, the sampling pitch of the signal trains of the second A image As_2 and the third A image As_3 is twice the sampling pitch of the signal train of the first A image As_1. The same applies to the B image. Therefore, the defocus conversion coefficient for multiplying the shift amounts dk2 and dk3 is twice the defocus conversion coefficient for multiplying dk1.

The effect of noise on the amount of defocus is that the correlation of noise components between the A and B images is lower than that of the first A and B images, and the second A and B images and the third A and B images. The defocus amounts Def2 and Def3 calculated based on the above are smaller than Def1. On the other hand, in the second and third images A and B, the sampling pitch is twice that of the first images A and B, and the number of signals constituting the signal sequence is halved. Therefore, the variation of the defocus amounts Def2 and Def3 obtained based on the second and third images A and B tends to be larger than the variation of Def1. Therefore, the average value of the defocus amounts Def2 and Def3 may be calculated as the defocus amount Def2'based on the image signal pair having a low correlation of noise components to obtain the defocus amount with suppressed variation. Hereinafter, the case where the defocus amount Def2'is calculated will be described.

In S5045, the AF unit 42 selects one of the defocus amount Def1 and Def2'as the final defocus amount Def.

The defocus amount Def1 based on the first image signal pair (first A image As_1 and first B image Bs_1) has the following characteristics.
There is a correlation of noise components between the A image and the B image, and the noise correlation has a singular value when the shift amount is 0. Therefore, the ratio of the correlation amount of the noise component to the correlation amount of the A image and the B image is relatively high, as in the case where the contrast of the subject in the focus detection area is low or the environmental brightness at the time of shooting is low. In some situations, the defocus amount detection accuracy may decrease.
The first image signal pair is larger than the second image signal pair (second A image As_2 and second B image Bs_2) and the third image signal pair (A image As_3 and third B image Bs_3). The sampling frequency is high. Therefore, the difference between the spatial frequency band of the image signal used for focus detection and the spatial frequency band of the imaging signal is smaller than that of the defocus amount Def2'(and Def2, Def3), and the defocus amount is the aberration of the photographing optical system. Less susceptible to quantity. Therefore, the amount of defocus corresponding to the focusing position where the difference from the best focusing position of the image pickup signal is small can be detected.

On the other hand, the defocus amount Def2'based on the second image signal pair and the third image signal pair has the following characteristics. The correlation of noise components between the A image and the B image is low. Therefore, the ratio of the correlation amount of the noise component to the correlation amount of the A image and the B image is relatively high, as in the case where the contrast of the subject in the focus detection area is low or the environmental brightness at the time of shooting is low. Even in a situation, the detection accuracy of the defocus amount is unlikely to decrease.
The second and third image signal pairs have a lower sampling frequency than the first image signal pair. Therefore, the difference between the spatial frequency band of the image signal used for the focus detection and the spatial frequency band of the image pickup signal is larger than that of the defocus amount Def1. As a result, the difference between the focused position corresponding to the detected defocus amount and the best focused position of the imaging signal may be larger than Def1. The defocus amounts Def2 and Def3 also have the same characteristics.

Based on these characteristics of the defocus amounts Def1 and Def2', the AF unit 42 has a shooting environment in which, for example, the presence or absence of a correlation between the noise components between the A image and the B image affects the error of the detected defocus amount. Depending on whether or not, the final detection defocus amount can be selected.

In the present embodiment, the AF unit 42 selects the defocus amount according to the magnitude relationship between the peak value of the AF image signals (first to third A and B images) obtained in S503 and the predetermined threshold value PeakTh. do. Specifically, in S5045, the AF unit 42 determines whether or not the peak value is larger than the predetermined threshold value PeakTh, and if it is determined to be large, the process proceeds to S5046, and if it is not determined to be large, the process is performed to S5047. Proceed to. Here, as the threshold value PeakTh, a value stored in advance in the non-volatile memory 56 is used according to a combination of imaging conditions such as aperture, storage time (electronic shutter speed), and ISO sensitivity at the time of shooting, and the above-mentioned optical conditions. Can be done.
In S5046, the AF unit 42 selects Def1 and ends the process.
In S5047, the AF unit 42 selects Def2'and ends the process.

According to the present embodiment, as an image signal pair based on a plurality of types of signals read from the same pixel, a first image signal pair of a type having a large correlation of noise components included in the image signal pair and an image signal pair. Generates a second image signal pair whose correlation of noise components contained in is lower than that of the first image signal pair. Then, one of the defocus amount detected based on the first image signal pair and the defocus amount based on the second image signal pair, the detected defocus amount is easily affected by the correlation of the noise component. Select and use depending on whether it is a condition or not. With such a configuration, in the focus detection device that performs the focus detection of the phase difference detection method based on the signal pair obtained from the image sensor and its control method, the influence of the correlated noise contained in the signal pair on the focus detection. It can be suppressed.

(Modification example 1)
In the present embodiment, in S5045 to S5047, one of the plurality of defocus amounts detected based on the first to third image signal pairs is based on the peak value of the first to third image signal pairs. And selected. The selection based on the signal peak value is particularly effective when there is a negative correlation between the noise components contained in the A image and the B image. The noise component contained in the A image and the B image has a negative correlation when the noise component having a positive correlation, which is proportional to the signal amount of the A + B image and the A image, is small. In other words, it is a case where the signal amounts of the A image and the B image are relatively small. Therefore, in the present embodiment, the peak value is used as an evaluation value for determining whether or not the noise components contained in the A image and the B image have a negative correlation and the signal amounts of the A image and the B image are relatively small. board.

However, the essence of the determination of S5045 is the determination of whether or not the correlation of the noise components included in the image signal pair affects the phase difference detection error. Therefore, the same determination may be performed using other evaluation values. For example, instead of the peak value, the bottom value, amplitude value, metering value, and the like of the signal can be used as evaluation values to determine the case where the signal amounts of the A image and the B image are relatively small.

Further, when the signal amounts of the A image and the B image are relatively large, it is considered that there is a positive correlation between the noise components contained in the A image and the B image. Therefore, by selecting Def2', the noise component can be obtained. Focus detection that suppresses the influence of The relatively large amount of signals of the A image and the B image can be determined, for example, by the fact that the bottom value of the first to third image signal pairs is larger than the threshold value. However, it can also be determined by using other evaluation values such as, for example, the integrated value of the luminance outputs of the A image and the B image. This is because there is a correlation between the magnitude of the signal amount and the amount of noise as well as the bottom value of the signal. Therefore, in S5045, it is determined whether or not the bottom value of the first to third image signal pairs is larger than the threshold value, and if it is determined to be large, the process proceeds to S5047, and if it is not determined to be large, the process proceeds to S5046. May be good.

In addition, it may be configured to judge both the case where the signal amount of the A image and the B image is relatively small and the case where the signal amount is large. For example, when the peak value of the first to third image signal pairs is equal to or less than the first threshold value and when the bottom value of the first to third image signal pairs is larger than the second threshold value, Def'2 is set. It can be configured to select and otherwise select Def1. In this way, if it is determined that the correlation of the noise components contained in the image signal pair affects the phase difference detection error, Def'2 can be selected, and if it is not determined, Def1 can be selected. be.

(Modification 2)
In the present embodiment, the final defocus amount is selected from a plurality of defocus amounts detected from a plurality of image signal pairs. However, the defocus amount may be detected and used only from one image signal pair that has no correlation with the noise components contained in the plurality of image signal pairs or is sufficiently small. As a result, the calculation load required for calculating the defocus amount can be reduced. In this case, an image signal pair that is assumed to have no correlation or a sufficiently small noise component can be predetermined from a plurality of image signal pairs based on, for example, a generation method.

デジタルフィルタを適用した第１のＡ像（Ａｓ＿１）および第１のＢ像（Ｂｓ＿１）を構成する信号列は、偶数番目のＡ信号（Ｂ信号）で算出される信号と、奇数番目のＡ信号（Ｂ信号）で算出される信号とが交互に配列された信号列になる。そのため、式（１’）のように、Ｂ像を１つずらして相関量ＣＯＲを算出することにより、Ａ像とＢ像とにおけるノイズ成分の相関による影響を低減した相関量ＣＯＲを得ることができる。 (Modification example 3)
In the present embodiment, the second and third image signal pairs in which the row position of the pixel group used for generating the A image and the row position of the pixel group used for generating the B image are different from each other. By using the above, the influence of the correlation of the noise components contained in the A image and the B image on the defocus amount was suppressed. However, the same effect can be obtained by calculating the correlation amount COR after applying a digital filter to the first A image (As_1) and the first B image (Bs_1). For example, after applying a row-direction digital filter such as [1,0, -1] to the first A image (As_1) and the first B image (Bs_1), the following equation (1') is used. The correlation amount COR (k) may be calculated.

The signal sequences constituting the first A image (As_1) and the first B image (Bs_1) to which the digital filter is applied are a signal calculated by an even-numbered A signal (B signal) and an odd-numbered A signal. The signal calculated in (B signal) becomes a signal sequence in which the signals are arranged alternately. Therefore, by calculating the correlation amount COR by shifting the B image by one as in the equation (1'), it is possible to obtain the correlation amount COR in which the influence of the correlation of the noise components in the A image and the B image is reduced. can.

(Modification example 4)
In the present embodiment, the second and third image signal pairs in which the row position of the pixel group used for generating the A image and the row position of the pixel group used for generating the B image are different from each other. By using the above, the influence of the correlation of the noise components contained in the A image and the B image on the defocus amount was suppressed. This is because it is assumed that the noise contained in the outputs of a plurality of pixels sharing the same microlens has a correlation. However, not only when the shared microlens is the same, the A image and the B image can be generated so as to eliminate or suppress an arbitrary factor having a correlation between the noise components contained in the A image and the B image. .. For example, the A signal and the B signal can be selected so that one or more of the floating diffusion, the signal output line, and the amplifier circuit existing in the signal path do not overlap. Further, when the transmittances of the color filters of the same color vary, the A signal and the B signal can be selected so as not to use the output of the pixel having the color filters having similar transmittances.

● (Second embodiment)
Next, a second embodiment of the present invention will be described. The main difference from the first embodiment is a method of generating second and third image signal pairs (As_2 and Bs_2, As_3 and Bs_3) having a low correlation of noise components. In the first embodiment, since the second and third image signal sequences are generated by thinning out the signals in the horizontal direction from one pixel row, the resolution of the phase difference detectable from the image signal pair may decrease. There is. In the second embodiment, signals of different pixel rows are used to generate second and third image signal pairs with low correlation of noise components without thinning out the signals in the horizontal direction. As a result, the second (or third) image signal has a resolution equivalent to the resolution of the phase difference that can be detected from the first image signal pair, and suppresses the influence of the correlation of noise components contained in the image signal. It makes it possible to detect the phase difference from the pair.

A block diagram of the image pickup device (FIG. 1), an explanatory diagram of the image sensor and the focus detection method (FIGS. 3 to 4), an explanatory diagram of the focus detection region (FIG. 6), a flowchart of the focus adjustment operation and the calculation of the defocus amount. (FIGS. 8 and 9) are also used in the present embodiment.

Hereinafter, the process of generating the focus detection signal in the second embodiment will be described.
FIG. 10 shows the pixels of 4 rows and 2N columns arranged in the focus detection area 401. Also in this embodiment, the pixel (photodiode 201a) used for creating the signal of the AF image A in the i-th row and j-th column and its output are represented by A (i, j). Similarly, the pixel (photodiode 201b) used to create the signal of the AF B image in the i-th row and j-th column and its output are represented by B (i, j).

In the first embodiment, the signal sequence is generated from the additional pixel signal obtained by adding the outputs of two pixels in the horizontal direction, but in the present embodiment, the added pixel signal obtained by adding the outputs of four pixels of horizontal 2 pixels × vertical 2 pixels. It is assumed that a signal sequence is generated from. Here, the individual signals constituting the first A image As_1 generated from the pixels in the i-th row are As_1 (i, k), and the individual signals constituting the first B image Bs_1 are Bs_1 (i, k). ) (K is the number of the signal constituting each signal sequence). In this case, As_1 (1, k) and Bs_1 (1, k) for the pixels of 4 rows and 2 N columns shown in FIG. 7 can be expressed by the following equation (15).
As_1 (i, k) = A (2 × i-1, 2 × (k-1) +1) + A (2 × i-1, 2 × (k-1) + 2) + A (2 × i, 2 × (k-1) +1) + A (2 × i, 2 × (k-1) + 2)
Bs_1 (i, k) = B (2 x i-1, 2 x (k-1) + 1) + B (2 x i-1, 2 x (k-1) + 2) + B (2 x i, 2 × (k-1) +1) + B (2 × i, 2 × (k-1) + 2)
(1 ≤ i ≤ 2) (1 ≤ k ≤ N) ... (15)

Further, the individual signals constituting the second A image As_2 generated from the pixels in the i-th row are As_2 (i, k), and the individual signals constituting the second B image Bs_2 are Bs_2 (i, k). And. In this case, As_2 (1, k) and Bs_2 (1, k) for the pixels of 4 rows and 2 N columns shown in FIG. 10 can be expressed by the following equation (16).
As_1 (i, k) = As_1 (i, k)
Bs_1 (i, k) = Bs_1 (i + 1, k)
(I = 1) (1 ≦ k ≦ N)… (16)

Further, the individual signals constituting the third A image As_3 generated from the pixels in the i-th row are As_3 (i, k), and the individual signals constituting the third B image Bs_3 are Bs_3 (i, k). And. In this case, As_3 (1, k) and Bs_3 (1, k) for the pixels of 4 rows and 2 N columns shown in FIG. 10 can be expressed by the following equation (17).
As_3 (i, k) = As_1 (i + 1, k)
Bs_3 (i, k) = Bs_1 (i, k)
(I = 1) (1 ≦ k ≦ N)… (17)

Also in this embodiment, the first A image (As_1) and the first B image (Bs_1) are generated based on the signals of the pixels sharing the microlens and the readout circuit. Therefore, the noise component contained in the A signal and the A + B signal has a positive correlation with the signal amount, and the noise component contained in the B signal generated by subtracting the A signal from the A + B signal is the noise contained in the A signal. Has a negative correlation with the components. Therefore, the noise components contained in the first A image (As_1) and the first B image (Bs_1) represented by the equation (15) have a correlation.

On the other hand, the second A image (As_2) and the second B image (Bs_2) represented by the equation (16) are the first A image (As_1) and the first image based on different pixel groups (pixel rows). It is a combination of B images (Bs_1). Therefore, the second A image (As_2) and the second B image (Bs_2) have a relationship in which the sampling position of the optical image of the subject is deviated. As described above, since the pixel group used for generating the second A image (As_2) and the pixel group used for generating the second B image (Bs_2) do not overlap, the second A image (As_2) The correlation between the noise component and the second B image (Bs_2) is low. Therefore, by calculating the correlation amount COR using the second A image (As_2) and the second B image (Bs_2), it is possible to obtain a correlation amount in which the influence of the correlation of the noise component is suppressed. .. These things also apply to the third image A (As_3) and the third image B (Bs_3) represented by the formula (17).

Further, the second and third image signal pairs of the present embodiment have a sampling pitch in the row direction equal to that of the first image signal pair. Therefore, unlike the first embodiment, the phase difference detected from the second image signal pair and the third image signal pair can also achieve the same resolution as the phase difference detected from the first image signal pair. ..

As described above, in the present embodiment, for each predetermined pixel group, based on the output signal of the pixel group, the first signal sequence pair (first A image As_1 and first A image As_1 and first) correlating with each other's noise components. B image Bs_1) of.
Further, based on the output signals of different pixel groups of the first signal sequence (first A image As_1) and the second signal sequence (first B image Bs_1) constituting the first signal sequence pair. The generated first signal sequence and the second signal sequence are combined to generate a second signal sequence pair.
Further, based on the output signals of different pixel groups of the first signal sequence (first A image As_1) and the second signal sequence (first B image Bs_1) constituting the first signal sequence pair. The generated first signal sequence and the second signal sequence are combined to generate a third signal sequence pair. Here, the combination of the first signal string and the second signal string is different between the second signal string pair and the third signal string pair.

For each of the three pairs of signal sequences generated as described above, the phase difference (shift amount) dk1 to dk3 is obtained in S5041 to 5044 of the defocus amount calculation process (FIG. 9) as in the first embodiment. The detection and defocus amounts Def1 and Def2'are calculated. Since the shift amounts dk2 and dk3 obtained in the present embodiment are based on the correlation amount of the images having different pixel rows, an error may occur due to the difference between the waveforms of the A image and the B image. However, since the error that can be included in the shift amount dk2 and the error that can be included in the shift amount dk3 have different signs and are about the same amount, the error is reduced by averaging Def2 and Def3 to Def2'. do.

As described above, according to the present embodiment, a pair of image signals used for calculating the correlation amount is generated based on the outputs of different pixel rows. By calculating the correlation amount based on this pair of image signals, it is possible to obtain a correlation amount in which the influence of the correlation of noise contained in the image signal is reduced. Further, the phase difference and the defocus amount can be calculated with the same resolution as the pair of image signals generated based on the signals of the same pixel row.

(Modification example)
In the second embodiment, two types of image signal pairs (first image signal pairs) in which the combination of the A image and the B image is different from the first image signal pair from a plurality of first image signal pairs generated from different pixel rows. The second and third image signal pairs) were generated and the correlation amount was calculated. However, the method of generating the second and third image signal pairs is not limited to this. For example, when calculating the correlation amount for a signal obtained by adding a plurality of A images (B images), the A image obtained from the odd-numbered pixel rows is added for the A image, and the even-numbered B image is added. The B image obtained from the pixel rows of may be added. On the contrary, for the A image, the A image obtained from the even-numbered pixel rows may be added, and for the B image, the B image obtained from the odd-numbered pixel rows may be added. Moreover, other method may be used.

Further, it is not necessary to perform the averaging process of the correlation amounts Def2 and Def3 calculated from the image signal pair having a low correlation of the noise components. For example, when it is expected that the difference between the waveforms of the A image and the B image is sufficiently small, only one of Def2 and Def3 may be obtained and used as Def2'. For example, the case where the contrast direction of the subject is not diagonal and the contrast changes only in the phase difference detection direction can be exemplified, but the present invention is not limited to this.

In order to reduce the difference in the shape of the image signal due to the generation of the image signal pair from different pixel rows, the B image paired with the A image generated from the output signal of the nth pixel row is generated (n-1). The two B images generated from the output signals of the th and (n + 1) th pixel rows may be averaged and generated. Further, instead of the averaging process of the correlation amounts Def2 and Def3, the averaging process may be performed on the shift amounts dk2 and dk3.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It is also possible to realize the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

14 ... image pickup element, 20 ... image processing unit, 50 ... system control unit, 100 ... camera, 300 ... photographing lens, 306 ... lens mount, 342 ... focus control unit, 346 ... lens system control unit.

Claims (20)

A plurality of first signals obtained from a plurality of first photoelectric conversion units that receive a light flux passing through the first pupil region of the exit pupil of the photographing optical system, and a second pupil of the exit pupil of the photographing optical system. A generation means for generating a plurality of image signal pairs from a plurality of second signals obtained from a plurality of second photoelectric conversion units that receive a light flux passing through a region.
For each of the plurality of image signal pairs, a calculation means for calculating the phase difference of the pair of image signals by a correlation calculation using a pair of image signals constituting the image signal pair.
A focus detection device including an adjusting means for adjusting the focusing distance of the photographing optical system based on the phase difference calculated by the calculating means.
Each of the plurality of image signal pairs comprises a first image signal and a second image signal.
The generation means generates the first image signal from the first signal and the second image signal from the second signal.
Of the plurality of image signal pairs, a second image signal pair is formed rather than a correlation of noise components contained in the first image signal and the second image signal forming the first image signal pair. The correlation between the noise component contained in the first image signal and the second image signal is low.
The adjusting means is based on the phase difference of the first image signal pair when the evaluation value of the contrast of the subject is equal to or more than the threshold value, and the second image signal when the evaluation value is less than the threshold value. based on the phase difference of the pair to adjust the focal distance of the photographing optical system,
A focus detector characterized by that. A plurality of first signals obtained from a plurality of first photoelectric conversion units that receive a light flux passing through the first pupil region of the exit pupil of the photographing optical system, and a second pupil of the exit pupil of the photographing optical system. A generation means for generating a plurality of image signal pairs from a plurality of second signals obtained from a plurality of second photoelectric conversion units that receive a light flux passing through a region.
A calculation means for calculating the phase difference of the pair of image signals by a correlation calculation using a pair of image signals constituting the image signal pair for each of the plurality of image signal pairs.
A focus detection device including an adjusting means for adjusting the focusing distance of the photographing optical system based on the phase difference calculated by the calculating means.
Each of the plurality of image signal pairs comprises a first image signal and a second image signal.
The generation means generates the first image signal from the first signal and the second image signal from the second signal.
Of the plurality of image signal pairs, a second image signal pair is formed rather than a correlation of noise components contained in the first image signal and the second image signal forming the first image signal pair. The correlation between the noise component contained in the first image signal and the second image signal is low.
The adjusting means is based on the phase difference of the first image signal pair when the evaluation value of the environmental brightness is equal to or more than the threshold value, and the second image signal pair when the evaluation value is less than the threshold value. The focusing distance of the photographing optical system is adjusted based on the phase difference of.
A focus detector characterized by that. A plurality of first signals obtained from a plurality of first photoelectric conversion units that receive a light flux passing through the first pupil region of the exit pupil of the photographing optical system, and a second pupil of the exit pupil of the photographing optical system. A generation means for generating a plurality of image signal pairs from a plurality of second signals obtained from a plurality of second photoelectric conversion units that receive a light flux passing through a region.
A calculation means for calculating the phase difference of the pair of image signals by a correlation calculation using a pair of image signals constituting the image signal pair for each of the plurality of image signal pairs.
A focus detection device including an adjusting means for adjusting the focusing distance of the photographing optical system based on the phase difference calculated by the calculating means.
Each of the plurality of image signal pairs comprises a first image signal and a second image signal.
The generation means generates the first image signal from the first signal and the second image signal from the second signal.
Of the plurality of image signal pairs, a second image signal pair is formed rather than a correlation of noise components contained in the first image signal and the second image signal forming the first image signal pair. The correlation between the noise component contained in the first image signal and the second image signal is low.
The adjusting means is based on the phase difference of the first image signal pair when the peak value of the plurality of image signal pairs is larger than the threshold value, and the second adjustment means when the peak value is not larger than the threshold value. The focusing distance of the photographing optical system is adjusted based on the phase difference of the image signal pair.
A focus detector characterized by that. A plurality of first signals obtained from a plurality of first photoelectric conversion units that receive a light flux passing through the first pupil region of the exit pupil of the photographing optical system, and a second pupil of the exit pupil of the photographing optical system. A generation means for generating a plurality of image signal pairs from a plurality of second signals obtained from a plurality of second photoelectric conversion units that receive a light flux passing through a region.
A calculation means for calculating the phase difference of the pair of image signals by a correlation calculation using a pair of image signals constituting the image signal pair for each of the plurality of image signal pairs.
A focus detection device comprising an adjusting means for adjusting the focusing distance of the photographing optical system based on the phase difference.
Each of the plurality of image signal pairs comprises a first image signal and a second image signal.
The generation means generates the first image signal from the first signal and the second image signal from the second signal.
Of the plurality of image signal pairs, a second image signal pair is formed rather than a correlation of noise components contained in the first image signal and the second image signal forming the first image signal pair. The correlation between the noise component contained in the first image signal and the second image signal is low.
The adjusting means is said to be the first when it is determined that the signal amount of the plurality of image signal pairs is large based on the magnitude relationship between the evaluation value and the threshold value regarding the amount of signals of the plurality of image signal pairs. If it is not determined that the signal amount of the plurality of image signal pairs is large based on the phase difference of the two image signal pairs, the focusing of the photographing optical system is based on the phase difference of the first image signal pair. Adjust the distance,
A focus detector characterized by that. The fourth aspect of claim 4, wherein the evaluation value is a peak value of the plurality of image signal pairs, a bottom value of the plurality of image signal pairs, an amplitude value of the plurality of image signal pairs, or a photometric value. Focus detector. The generation means
A part of the first signal used for generating the first image signal of the first image signal pair is used to generate the first image signal of the second image signal pair.
A part of the second signal used for generating the second image signal of the first image signal pair is used to generate the second image signal of the second image signal pair.
The focus detection device according to any one of claims 1 to 5, wherein the focus detection device is characterized. The plurality of image signal pairs further include a third image signal pair.
The first image signal forming the third image signal pair is more than the correlation of the noise component contained in the first image signal and the second image signal forming the first image signal pair. The correlation of the noise component contained in the second image signal is low,
The generation means
It is a part of the first signal used for generating the first image signal of the first image signal pair, and is used for generating the first image signal of the second image signal pair. The first image signal of the third image signal pair is generated using the first signal that does not exist.
It is a part of the second signal used for generating the second image signal of the first image signal pair, and is used for generating the second image signal of the second image signal pair. The second image signal of the third image signal pair is generated by using the second signal that does not exist.
When the adjusting means adjusts the focusing distance of the photographing optical system without being based on the phase difference of the first image signal pair, the phase difference of the second image signal pair and the third image signal pair The focusing distance of the photographing optical system is adjusted based on the phase difference of
The focus detection device according to any one of claims 1 to 6, wherein the focus detection device is characterized. The generation means
The second image signal pair is generated as a combination of a first image signal and a second image signal different from the first image signal pair.
The focus detection device according to any one of claims 1 to 5, wherein the focus detection device is characterized. The plurality of image signal pairs further include a third image signal pair.
The first image signal forming the third image signal pair is more than the correlation of the noise component contained in the first image signal and the second image signal forming the first image signal pair. The correlation of the noise component contained in the second image signal is low,
The generation means
The third image signal pair is generated as a combination of the first image signal and the second image signal constituting the different first image signal pair.
In the second image signal pair and the third image signal pair, the combination of the first image signal and the second image signal constituting the first image signal pair is different .
When the adjusting means adjusts the focusing distance of the photographing optical system without being based on the phase difference of the first image signal pair, the phase difference of the second image signal pair and the third image signal pair The focusing distance of the photographing optical system is adjusted based on the phase difference of
The focus detection device according to claim 8. When the bottom value of the plurality of image signal pairs is determined to be larger than the threshold value, the adjusting means uses a defocus amount based on an image signal pair different from that of the first image signal pair to obtain the photographing optical system. The focus detection device according to claim 4 or 5 , wherein the focusing distance of the device is adjusted. Each of the plurality of first signals and the plurality of second signals has a microlens and a first photoelectric conversion unit and a second photoelectric conversion unit that share the microlens. Obtained from the imaging element,
The image signal pair is a pair of the plurality of first signals and the plurality of second signals obtained from the plurality of first photoelectric conversion units and the plurality of second photoelectric conversion units that do not share a microlens. The focus detection device according to any one of claims 1 to 10, wherein the focus detection device is included. The image signal pair includes a plurality of the first signals and a plurality of the first signals obtained from the plurality of the first photoelectric conversion units and the plurality of the second photoelectric conversion units that share the microlens or the signal path. It is composed of a first image signal and a second image signal having a lower correlation of noise components contained than the first image signal and the second image signal generated from the second signal.
The focus detection device according to claim 11. The second signal is obtained from the first photoelectric conversion unit from the third signal obtained from both the first photoelectric conversion unit and the second photoelectric conversion unit that share the microlens. The focus detection device according to any one of claims 1 to 12 , wherein the focus detection device is obtained by subtracting the first signal. The plurality of the second signals are obtained by taking the difference between the plurality of the first signals and the plurality of third signals obtained from the first photoelectric conversion unit and the second photoelectric conversion unit. The focus detection device according to any one of claims 1 to 13, wherein the focus detection device is obtained. The focus detection device according to any one of claims 1 to 14.
An image pickup device having the plurality of the first photoelectric conversion units and the plurality of the second photoelectric conversion units.
An imaging device characterized by having. The generation means includes a plurality of first signals obtained from a plurality of first photoelectric conversion units that receive a light flux passing through the first pupil region of the exit pupil of the photographing optical system, and the exit pupil of the photographing optical system. A generation step of generating a plurality of image signal pairs from a plurality of second signals obtained from a plurality of second photoelectric conversion units that receive a light flux passing through the second pupil region.
A calculation step in which the calculation means calculates the phase difference of the pair of image signals by a correlation calculation using a pair of image signals constituting the image signal pair for each of the plurality of image signal pairs.
The adjusting means is a control method of a focus detecting device having an adjusting step of adjusting the focusing distance of the photographing optical system based on the phase difference calculated in the calculation step.
Each of the plurality of image signal pairs comprises a first image signal and a second image signal.
In the generation step, the first image signal is generated from the first signal, and the second image signal is generated from the second signal.
Of the plurality of image signal pairs, a second image signal pair is formed rather than a correlation of noise components contained in the first image signal and the second image signal forming the first image signal pair. The correlation between the noise component contained in the first image signal and the second image signal is low.
In the adjustment step, the second image signal is based on the phase difference of the first image signal pair when the evaluation value of the contrast of the subject is equal to or more than the threshold value, and when the evaluation value is less than the threshold value. based on the phase difference of the pair to adjust the focal distance of the photographing optical system,
A control method for a focus detector, characterized in that. The generation means includes a plurality of first signals obtained from a plurality of first photoelectric conversion units that receive a light flux passing through the first pupil region of the exit pupil of the photographing optical system, and the exit pupil of the photographing optical system. A generation step of generating a plurality of image signal pairs from a plurality of second signals obtained from a plurality of second photoelectric conversion units that receive a light flux passing through the second pupil region.
A calculation step in which the calculation means calculates the phase difference of the pair of image signals by a correlation calculation using a pair of image signals constituting the image signal pair for each of the plurality of image signal pairs.
The adjusting means is a control method of a focus detecting device having an adjusting step of adjusting the focusing distance of the photographing optical system based on the phase difference calculated in the calculation step.
Each of the plurality of image signal pairs comprises a first image signal and a second image signal.
In the generation step, the first image signal is generated from the first signal, and the second image signal is generated from the second signal.
Of the plurality of image signal pairs, a second image signal pair is formed rather than a correlation of noise components contained in the first image signal and the second image signal forming the first image signal pair. The correlation between the noise component contained in the first image signal and the second image signal is low.
In the adjustment step, when the evaluation value of the environmental brightness is equal to or more than the threshold value, the phase difference of the first image signal pair is used, and when the evaluation value is less than the threshold value, the second image signal pair is obtained. The focusing distance of the photographing optical system is adjusted based on the phase difference of.
A control method for a focus detector, characterized in that. The generation means includes a plurality of first signals obtained from a plurality of first photoelectric conversion units that receive a light flux passing through the first pupil region of the exit pupil of the photographing optical system, and the exit pupil of the photographing optical system. A generation step of generating a plurality of image signal pairs from a plurality of second signals obtained from a plurality of second photoelectric conversion units that receive a light flux passing through the second pupil region.
A calculation step in which the calculation means calculates the phase difference of the pair of image signals by a correlation calculation using a pair of image signals constituting the image signal pair for each of the plurality of image signal pairs.
The adjusting means is a control method of a focus detecting device having an adjusting step of adjusting the focusing distance of the photographing optical system based on the phase difference calculated in the calculation step.
Each of the plurality of image signal pairs comprises a first image signal and a second image signal.
In the generation step, the first image signal is generated from the first signal, and the second image signal is generated from the second signal.
Of the plurality of image signal pairs, a second image signal pair is formed rather than a correlation of noise components contained in the first image signal and the second image signal forming the first image signal pair. The correlation between the noise component contained in the first image signal and the second image signal is low.
In the adjustment step, when the peak value of the plurality of image signal pairs is larger than the threshold value, the phase difference of the first image signal pair is used, and when the peak value is not larger than the threshold value, the second image signal pair is described. The focusing distance of the photographing optical system is adjusted based on the phase difference of the image signal pair.
A control method for a focus detector, characterized in that. The generation means includes a plurality of first signals obtained from a plurality of first photoelectric conversion units that receive a light flux passing through the first pupil region of the exit pupil of the photographing optical system, and the exit pupil of the photographing optical system. A generation step of generating a plurality of image signal pairs from a plurality of second signals obtained from a plurality of second photoelectric conversion units that receive a light flux passing through the second pupil region.
A calculation step in which the calculation means calculates the phase difference of the pair of image signals by a correlation calculation using a pair of image signals constituting the image signal pair for each of the plurality of image signal pairs.
The adjusting means is a control method of a focus detecting device having an adjusting step of adjusting the focusing distance of the photographing optical system based on the phase difference calculated in the calculation step.
Each of the plurality of image signal pairs comprises a first image signal and a second image signal.
In the generation step, the first image signal is generated from the first signal, and the second image signal is generated from the second signal.
Of the plurality of image signal pairs, a second image signal pair is formed rather than a correlation of noise components contained in the first image signal and the second image signal forming the first image signal pair. The correlation between the noise component contained in the first image signal and the second image signal is low.
In the adjustment step, when it is determined that the signal amount of the plurality of image signal pairs is large based on the magnitude relationship between the evaluation value and the threshold value regarding the large amount of signals of the plurality of image signal pairs, the first step is made. If it is not determined that the signal amount of the plurality of image signal pairs is large based on the phase difference of the two image signal pairs, the focusing of the photographing optical system is based on the phase difference of the first image signal pair. Adjust the distance,
A control method for a focus detector, characterized in that. A program for causing a computer included in a focus detection device to function as each means of the focus detection device according to any one of claims 1 to 14.

Images