JP7332302B2 - Imaging device and its control method - Google Patents

Description

The present invention relates to an imaging device and its control method.

In recent years, in addition to simply outputting an image signal photoelectrically converted by a pixel in an image pickup device, for example, techniques for expanding the dynamic range and outputting distance information to a subject have been proposed. Japanese Patent Application Laid-Open No. 2002-200002 proposes a technique that has a function of switching the input capacitance of an amplifier circuit provided for each column of an image sensor and switches the gain according to the signal level. With the configuration of switching gains as in Patent Document 1, image signals of a low gain signal and a high gain signal are output, and by synthesizing them in subsequent image processing, an image signal with a high dynamic range and low noise is obtained. can be created.

On the other hand, a so-called imaging plane phase difference focus detection method has been proposed, in which a pair of parallax images are read out from an imaging device and focus detection is performed using a phase difference detection method. As an example of an imaging device that outputs a signal that can be used for a focus detection method using an imaging surface phase difference method, a pair of photoelectric conversion units are provided for each microlens that constitutes a two-dimensionally arranged microlens array. there is something Japanese Patent Application Laid-Open No. 2002-200000 discloses an image pickup device capable of arbitrarily performing addition/non-addition of signals output from a pair of photoelectric conversion units to which light is incident via one microlens for each pair of photoelectric conversion units. A device has been proposed.

JP-A-2005-175517 JP-A-2001-83407

However, since the reading method of the high-gain image signal and the low-gain image signal for dynamic range expansion in Patent Document 1 is different from the reading method of the phase difference detection image signal in Patent Document 2, they are read in the same frame. could not be read by

Also, in order to maintain a high frame rate, it is assumed that driving for reading out the image signal for dynamic range expansion and driving for reading out the image signal for phase difference detection are switched in units of readout rows of the image sensor within one frame. In that case, the dynamic range cannot be expanded in the row from which the image signal for phase difference detection is read.

SUMMARY OF THE INVENTION The present invention has been made in view of the above-mentioned problems, and is intended to read out image signals necessary for focus detection and/or dynamic range expansion in a shorter time according to the subject to be photographed and the use of the read image signals. aim.

In order to achieve the above object, the imaging device of the present invention includes: a pixel region having a plurality of microlenses arranged in a matrix and a plurality of photoelectric conversion units configured for each microlens; Amplification means capable of amplifying a signal output from from using a plurality of different gains including at least a first gain, a partial signal that is a part of the signals of the plurality of photoelectric conversion units, and the plurality of photoelectric conversion units an imaging device having a scanning means capable of scanning the pixel region so as to read out an added signal obtained by adding the signals of; a control means for controlling the imaging device ; and using the plurality of different gains processing means for expanding a dynamic range using the added signal amplified by the above-described method ; and focus detection means for performing phase-difference focus detection using the partial signal and the added signal; The control means expands the dynamic range by the processing means, and uses the partial signal and the addition signal amplified using a plurality of different gains to perform phase-difference focus detection by the focus detection means. controlling the amplifying means to amplify the partial signal and the sum signal using the plurality of different gains when the first mode is set, and expanding the dynamic range by the processing means; and a second mode is set in which phase-difference focus detection is performed by the focus detection means using the partial signal and the sum signal amplified using the first gain. and controlling the amplifying means to amplify the partial signal and the sum signal using the first gain, expanding the dynamic range by the processing means, and performing phase-difference focusing by the focus detection means. controlling the scanning means so as not to read out the partial signal from the pixel area when a third mode in which detection is not performed is set, and amplifying the addition signal using the plurality of different gains; the dynamic range is expanded by the processing means, and the position is determined by the focus detection means using the partial signal and the sum signal amplified by using the first gain. When a fourth mode of phase difference focus detection is set, the partial signal is amplified using the first gain, and the sum signal is amplified using the plurality of different gains. to control the amplifying means ;

According to the present invention, image signals required for focus detection and/or dynamic range expansion can be read out in a shorter period of time depending on the subject to be photographed and the application of the read image signals.

1 is a diagram showing a schematic configuration of an image sensor according to an embodiment of the present invention; FIG. (a) A diagram showing details from a unit pixel of an image sensor to an AD circuit group, (b) A circuit diagram showing a configuration of a column amplifier. 4 is a timing chart showing control of column amplifiers when reading partial signals for phase difference detection and image signals for dynamic range expansion according to the embodiment of the present invention. 4 is a timing chart showing control of column amplifiers when reading partial signals for phase difference detection and image signals when dynamic range expansion according to the embodiment of the present invention is not performed. 4 is a timing chart showing control of column amplifiers when reading image signals for dynamic range expansion without reading partial signals for phase difference detection according to the embodiment of the present invention. FIG. 4 is a diagram showing readout timing of image signals from an image sensor in the first embodiment; FIG. 4 is a diagram showing readout timing of image signals from an image sensor in the first embodiment; 1 is a block diagram showing a schematic configuration of an imaging device according to a first embodiment; FIG. 4 is a flowchart showing readout control of the image sensor in the first embodiment; FIG. 10 is a diagram showing readout timing of image signals from an image sensor in the second embodiment; FIG. 2 is a block diagram showing a schematic configuration of an imaging device according to a second embodiment; FIG. FIG. 10 is a diagram showing image data in each block in the imaging device according to the second embodiment; FIG. 11 is a block diagram showing a schematic configuration of an imaging device according to a third embodiment; FIG. 9 is a flowchart showing processing in the third embodiment; FIG. 11 is a block diagram showing a schematic configuration of an imaging device according to a fourth embodiment; 10 is a flowchart showing processing in the fourth embodiment; FIG. 11 is a diagram showing details from a unit pixel of an image sensor to an AD circuit group in a fifth embodiment; FIG. 11 is a timing chart showing control of column amplifiers when partial signals are read in parallel at high gain and low gain from two photoelectric conversion elements according to the fifth embodiment;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings.

<First Embodiment>
FIG. 1(a) is a diagram showing a configuration example of an imaging device mounted with an AD converter according to an embodiment of the present invention. In the pixel region 100, a plurality of unit pixels 101 formed of photodiodes for photoelectric conversion or the like are arranged in a matrix. For phase difference detection, the unit pixel 101 includes a photoelectric conversion unit A and a photoelectric conversion unit B for one microlens 111, which will be described later. The focus can be detected by obtaining the phase difference of .

FIG. 1B is a conceptual diagram showing a cross section of the unit pixel 101. Under one microlens 111, two photoelectric conversion units A and B each having a photodiode are configured. It is shown that. Each unit pixel 101 is also provided with a color filter 112 . In general, one of the three colors of R (red), G (green), and B (blue) is often a Bayer array RGB primary color filter corresponding to each pixel, but this is not always the case. .

The vertical scanning circuit 102 performs timing control for sequentially reading pixel signals accumulated in the photoelectric conversion units A and B of the pixel region 100 during one frame period. In general, pixel signals are sequentially read out row by row from the upper row to the lower row during one frame period. In the present embodiment, from each unit pixel 101, in the vertical scanning circuit 102, the partial signal (A signal) which is the signal of the photoelectric conversion unit A and the addition signal (A+B signal) obtained by adding the signals of the photoelectric conversion unit A and the photoelectric conversion unit B are added. signal) is read out. By reading out in this way, the A+B signal can be used as it is as an image signal, and the B signal can be obtained by subtracting the A signal from the A+B signal, and focus detection by the imaging plane phase difference method can be performed. . However, if focus detection is not performed using the imaging plane phase difference method, only the A+B signal can be read out.

A column amplifier group 103 is composed of a plurality of column amplifiers configured for each column of the pixel region 100 and used to electrically amplify signals read out from the pixel region 100 . By amplifying the signal with the column amplifier group 103, the pixel signal level can be amplified against the noise generated in the AD circuit group 104 in the subsequent stage, and the S/N ratio can be equivalently improved. Note that the column amplifier group 103 can amplify signals using a plurality of gains, and in the present embodiment, by synthesizing signals amplified with different gains, the dynamic range is expanded. A detailed configuration of each column amplifier will be described later with reference to FIG.

The AD circuit group 104 is composed of a plurality of circuits configured for each column of the pixel region 100, and converts signals amplified by the column amplifier group 103 into digital signals. The pixel signals converted into digital signals are sequentially read by the horizontal transfer circuit 105 and input to the signal processing unit 106 . The signal processing unit 106 is a circuit that performs digital signal processing, and in addition to performing offset correction such as FPN correction by digital processing, it can easily perform gain calculation by performing shift calculation and multiplication. After performing each process, the image is output to the outside of the image sensor.

The memory 107 has a function of temporarily holding A signal, A+B signal, etc. read from the pixel region 100 and processed by the column amplifier group 103, the AD circuit group 104, and the signal processing unit 106. FIG.

In the example shown in FIG. 1B, each unit pixel 101 has two photoelectric conversion units A and B for one microlens 111. However, the number of photoelectric conversion units is The number is not limited to two, and may be more. Also, the pupil division direction may be horizontal or vertical, or may be mixed. Further, a plurality of pixels having different opening positions of the light receiving portion with respect to the microlens 111 may be provided. In other words, any configuration may be used as long as two signals for phase difference detection, such as the A signal and the B signal, are obtained as a result. Further, the present invention is not limited to a configuration in which all pixels have a plurality of photoelectric conversion units, and may be configured to discretely provide pixels as shown in FIG. 2 in normal pixels constituting an image sensor. Also, the same image sensor may include a plurality of types of pixels divided by different division methods.

Next, the circuit configuration and signal flow from the unit pixel 101 to the AD circuit group 104 will be described with reference to FIG. A photoelectric conversion element 1101 corresponding to the photoelectric conversion unit A in FIG. 1B and a photoelectric conversion element 1102 corresponding to the photoelectric conversion unit B in FIG. 1B share a microlens and perform photoelectric conversion. converts light into electric charge. The transfer switch 1103 transfers the charge generated by the photoelectric conversion element 1101 to the circuit in the subsequent stage, and the transfer switch 1104 transfers the charge generated by the photoelectric conversion element 1102 to the circuit in the subsequent stage. The charge holding unit 1105 temporarily holds charges transferred from the photoelectric conversion elements 1101 and 1102 when the transfer switches 1103 and 1104 are ON. Therefore, the charge holding portion 1105 can hold only the charge of either the photoelectric conversion element 1101 or the photoelectric conversion element 1102 or the sum of the charges of both the photoelectric conversion elements 1101 and 1102 . be. The pixel amplifier 1106 converts the charge held in the charge holding unit 1105 into a voltage signal, and transmits the voltage signal through the vertical output line 1113 to the subsequent column amplifier 103i. A current control unit 1107 controls the current of the vertical output line 1113 .

As described above, the column amplifier group 103 shown in FIG. 1 is composed of a plurality of column amplifiers 103i configured for each column, and amplifies the signal output to each vertical output line 1113 to produce an output signal of the AD circuit group 104 in the subsequent stage. output to Each AD circuit 104i forming the AD circuit group 104 converts analog signals output from the column amplifiers 103i of the same column into digital signals.

In the AD circuit 104i, the digital signal converted by the A/D converter 1109 is temporarily held in the memories 1110 and 1111. FIG. The memory 1110 stores pixel signals read from the photoelectric conversion element 1101 or the photoelectric conversion element 1102, and noise signals from a readout circuit portion (for convenience, the circuit from the charge holding portion 1105 to the A/D conversion portion 1109). and hold. On the other hand, the memory 1111 holds the noise signal of the readout circuit portion. Then, the subtraction unit 1112 subtracts the data held in the memory 1111 from the data held in the memory 1110 and outputs the result to the horizontal transfer circuit 105 as a pixel signal.

FIG. 2B is a diagram showing the configuration of the column amplifier 103i. The column amplifier 103i is an inverting amplifier circuit composed of an operational amplifier 207, capacitors 202 and 203 on the input side, and feedback capacitors 205 and 206. FIG. In addition, each switch 200, 201, 204 can switch the connection of the capacitors 202, 203, 205. FIG.

First, signals input from the unit pixel 101 are accumulated in the capacitors 202 and 203 with the switches 200 and 201 turned on. In the case of an image signal with proper exposure, the switches 201 and 204 are turned off and the switch 200 is turned on to apply a high gain to the image signal and read it out. Next, when reading out the image signal of the high luminance portion, the switch 200 is turned off and the switches 201 and 204 are turned on to apply a low gain to the image signal and read it. In this manner, by switching the capacitance of the capacitor with each switch, it is possible to apply different gains and read out the image signal. It is assumed that the dynamic range is expanded by synthesizing the image signals read out in this manner.

FIG. 3 is a timing chart showing control of the column amplifier 103i when reading the partial signal for phase difference detection and the image signal for expanding the dynamic range.

First, between times t1 and t4, the switch 200 is turned on and the switches 201 and 204 are turned off, thereby setting the gain of the column amplifier 103i to a high gain. In this state, the transfer switch 1103 is turned on at time t2 to read the A signal. At this time, the A signal read at high gain is A/D converted in the AD circuit 104i within the period from time t2 to t4.

Next, between times t4 and t5, the switch 200 is turned off and the switches 201 and 204 are turned on to set the gain of the column amplifier 103i to a low gain. At this time, the A signal read at the low gain is A/D converted in the AD circuit 104i within the period from time t4 to time t5.

Between times t5 and t8, the switch 200 is turned on again and the switches 201 and 204 are turned off to set the gain of the column amplifier 103i to a high gain. In this state, the transfer switch 1104 is turned on at time t6 to read the B signal. The A signal and the B signal are added in the charge holding unit 1105 and output as the A+B signal. At this time, the A+B signal read at the high gain is A/D converted in the AD circuit 104i within the period from time t6 to t8.

Between times t8 and t9, the switch 200 is turned off and the switches 201 and 204 are turned on to set the gain of the column amplifier 103i to a low gain. At this time, the A+B signal read at the low gain is A/D converted in the AD circuit 104i within the period from time t8 to t9.

FIG. 4 is a timing chart showing control of the column amplifier 103i when the partial signal for phase difference detection and the image signal are read when dynamic range expansion is not performed.

In this control, the gain of the column amplifier 103i is set to a high gain by turning on the switch 200 and turning off the switches 201 and 204 from time t11 to t16. In this state, the transfer switch 1103 is turned on at time t12 to read the A signal. At this time, the A signal read at high gain is A/D converted in the AD circuit 104i within the period from time t12 to t14.

Next, at time t14, the transfer switch 1104 is turned on to read the B signal. The A signal and the B signal are added in the charge holding unit 1105 and output as the A+B signal. At this time, the A+B signal read at the high gain is A/D converted in the AD circuit 104i within the period from time t14 to t16.

FIG. 5 is a timing chart showing control of the column amplifier 103i when reading the image signal for dynamic range expansion without reading the partial signal for phase difference detection.

In this control, the gain of the column amplifier 103i is set to a high gain by turning on the switch 200 and turning off the switches 201 and 204 from time t21 to t24. In this state, transfer switches 1103 and 1104 are turned on at time t22 to read out the A+B signal. At this time, the A+B signal read at the high gain is A/D converted in the AD circuit 104i within the period from time t22 to t24.

Next, between times t24 and t25, the switch 200 is turned off and the switches 201 and 204 are turned on to set the gain of the column amplifier 103i to a low gain. At this time, the A+B signal read at the low gain is A/D converted in the AD circuit 104i within the period from time t24 to t25.

FIG. 6 is a diagram showing readout timings of image signals from the image sensor in the first embodiment, and shows the concept of signals read out in each frame by the control shown in FIG. In FIG. 6, 1H transmission data refers to data for one row read out from the pixel area 100 .

In FIG. 6A, the image signal for phase difference detection, the image signal for phase difference detection when expanding the dynamic range, and the image signal for expanding the dynamic range are read for all lines within one frame period. FIG. 10 is a diagram showing read timing when Specifically, the A signal (hereinafter referred to as "high gain A signal") is read out with a high gain, and then the A signal (hereinafter referred to as "low gain A signal") is read out with a low gain. Furthermore, after reading the A+B signal (hereinafter referred to as "high gain A+B signal") with high gain, the A+B signal (hereinafter referred to as "low gain A+B signal") is read with low gain.

The high gain A signal thus read, the high gain B signal obtained by subtracting the high gain A signal from the high gain A+B signal, the low gain A signal, and the low gain A signal obtained by subtracting the low gain A signal from the low gain A+B signal are Using the obtained low-gain B signal, phase difference detection can be performed. Also, as the image signal for expanding the dynamic range, the high gain A+B signal and the low gain A+B signal can be used as they are.

As described above, in the configuration of the image sensor in the present embodiment, the reading method from the image sensor when reading data for one row can be changed. Image signals for enlargement can be read out.

However, if the image signal for phase difference detection and the image signal for dynamic range expansion are read in the same frame as shown in FIG. 6A, the transmission time for one frame becomes long. Therefore, if you want to quickly adjust the focus without recording, for example, when the shutter is half-pressed before shooting a still image, or when the screen is enlarged to adjust the focus, use the phase difference detection mode.

FIG. 6B is a diagram showing the readout timing when only the image signal for phase difference detection is read out in the mode for phase difference detection, and shows the concept of the signal read out in each frame by the control shown in FIG. ing. Through this control, the high gain A signal and the high gain A+B signal are read. By increasing the frame rate in this way, the information for phase difference detection can be preferentially acquired.

Also, when the focus is determined in advance and no focus information is required, such as when photographing a still image, the dynamic range expansion mode is selected. FIG. 7A is a diagram showing the readout timing when only the image signal for dynamic range expansion is read out in the dynamic range expansion mode, and shows the concept of the signal read out in each frame by the control shown in FIG. ing. With this control, the high gain A+B signal and the low gain A+B signal are read. By doing so, the imaging time for one frame can be shortened.

Further, if phase difference detection on the high luminance side is not required and only phase difference detection near proper exposure is required, a method of not outputting the A signal read out with a low gain is conceivable in order to increase the frame rate. FIG. 7(b) is a diagram showing read timing in such a case, in which the high gain A signal, high gain A+B signal, and low gain A+B signal are read. By doing so, it is possible to obtain an image signal for phase difference detection near proper exposure and an image signal for dynamic range expansion.

In addition to the above, the mode may be changed depending on the subject. For example, even in the dynamic range expansion mode of FIG. 7(a), if the subject to be photographed does not have a high-brightness subject, the mode is changed to the phase difference detection mode ( FIG. 6(b)) to capture a still image. Phase difference detection may also be performed.

The order of reading the high gain A signal, low gain A signal, high gain A+B signal, and low gain A+B signal is not limited to the example described above. For example, you can reverse the order of reading the low gain signal and the high gain signal, read the high gain signal first, then read the low gain signal, or read the low gain signal first, then read the high gain signal. It can be changed as appropriate, such as reading.

FIG. 8 is a block diagram showing the configuration of the imaging apparatus according to this embodiment, showing only the components directly related to the present invention. The signal flow when the image signal for phase difference detection and the image signal for dynamic range expansion are output in the same frame from the imaging device 400 as described with reference to FIG. 6A will be described below.

The image pickup device 400 is the image pickup device described with reference to FIG. . The distributor 401 distributes signals used as an image (low gain A+B signal, high gain A+B signal) and signals used for phase difference detection (low gain A signal , low gain A+B signal, high gain A signal, high gain A+B signal ) are output separately.

B signal generation section 408 subtracts high gain A from the high gain A+B signal output from distributor 401 to generate a high gain B signal. A low gain B signal is also generated by subtracting the low gain A signal from the low gain A+B signal.

The phase difference detection unit 403 detects the phase difference from the phase difference detection signals (high gain A signal, high gain B signal, low gain A signal, low gain B signal) output from the B signal generation unit 408. . A focus calculation unit 404 performs focus calculation based on the phase difference information detected here and the focus position of the lens. Based on the obtained focus information, the imaging apparatus notifies the user of the focus information and performs focus control of the lens.

When the high-gain A+B signal is saturated, the image synthesizing unit 402 synthesizes the dynamic range expansion image using an arbitrary synthesizing method from the signal for dynamic range expansion output from the image sensor. As an example, there is a method of synthesizing a dark portion of an object using a high-gain image and a bright portion using a low-gain image. For example, the synthesizing algorithm is not limited. A control unit 405 performs switching control of the shutter speed and gain of the image pickup device 400, change of read driving, and the like.

In the above example, the signal flow when the low gain A signal, the high gain A signal, the low gain A+B signal, and the high gain A+B signal are read has been described. , the signal may be read out as described in FIG. 7(b). In this case, the image signal for phase difference detection and the image signal for dynamic range expansion may be distributed by the distributor 401 to the image synthesizing unit 402 and the phase difference detecting unit 403 and output as necessary.

FIG. 9 is a flowchart showing readout control of the image sensor 400 by the control unit 405 in the first embodiment. First, in S100, the control unit 405 determines whether expansion of the dynamic range is necessary. It should be noted that the determination as to whether or not the dynamic range needs to be expanded is made according to, for example, the half-pressed state of the shutter before shooting a still image or the shooting mode set by the user, as described above. Alternatively, the determination result of whether or not the high-gain A+B signal of the image signal of the previous frame by the image synthesizing unit 402 is saturated may be used for determination.

If expansion of the dynamic range is not required, the process proceeds to S112 to read the high gain A signal for phase difference detection, and then proceeds to S113 to read the high gain A+B signal. Then, in S121, it is determined whether reading from all rows has been completed. If not, the process returns to S112 to continue reading. This corresponds to the readout order in the phase difference detection mode shown in FIG. 6(b).

On the other hand, if expansion of the dynamic range is required, the process advances to S101 to determine whether phase difference detection is required. If it is determined that there is no need, the process proceeds to S110 to read the low gain A+B signal, and then proceeds to S111 to read the high gain A+B signal. Then, in S122, it is determined whether reading from all rows has been completed. If not, the process returns to S110 to continue reading. This corresponds to the read order in the dynamic range expansion mode shown in FIG. 7(a).

On the other hand, if phase difference detection is required, the process advances to S102 to determine whether or not phase difference detection of the high-luminance portion is required. If it is determined that there is no need, the high gain A signal is read in S107, the high gain A+B signal is read in S108, and the low gain A+B signal is read in S109. Then, in S123, it is determined whether reading from all rows has been completed. If not, the process returns to S107 to continue reading. This corresponds to the reading order shown in FIG. 7(b).

If the phase difference detection of the high-brightness portion is also required, the process proceeds to S103. Then, the high gain A signal is read in S103, the high gain A+B signal in S104, the low gain A signal in S105, and the low gain A+B signal in S106 . Then, in S124, it is determined whether reading from all rows has been completed. If not, the process returns to S103 to continue reading. This corresponds to the reading order shown in FIG.

When the readout of one frame is completed by one of the above readout methods, the processing in FIG. 9 ends.

As described above, according to this embodiment, an image signal used for dynamic range expansion and an image signal used for phase difference detection can be obtained in each frame. In addition, by controlling so as not to read out unnecessary image signals, the frame rate can be increased compared to the case where all the image signals used for dynamic range expansion and the image signals used for phase difference detection are read out from each row.

<Second embodiment>
Next, a second embodiment of the invention will be described. Note that the imaging element in the second embodiment is the same as that described in the first embodiment, so description thereof will be omitted here.

FIG. 10 is a diagram showing readout timings of image signals from the image sensor in the second embodiment. As in FIG. 6, 1H transmission data refers to data for one row read out from the pixel area 100 .

FIG. 10A shows readout timing when the image signal for phase difference detection and the image signal for dynamic range expansion are read for all lines within one frame period without performing phase difference detection on the high luminance side. , which is the reading method shown in FIG. 7(b).

In this way, if image signals for phase difference detection and dynamic range expansion are read out in addition to normal image signals for all lines, the amount of data will be less than normal readout without dynamic range expansion and phase difference detection. is tripled, putting pressure on the transmission band. As a result, the frame rate becomes slower than when only normal image signals are read.

Therefore, in this embodiment, as shown in FIG. 10B, the image signal for phase difference detection and the image signal for dynamic range expansion are alternately output for each row in order to increase the frame rate. By reducing the data amount of the image signal of one frame in this manner, the frame rate can be increased. Further, with the readout method shown in FIG. 10B, phase difference detection is possible in the entire area of the screen. Therefore, even when the user wants to focus on an arbitrary place, the focus can be controlled with high accuracy. It is possible.

FIG. 11 is a block diagram showing a schematic configuration of an imaging device according to the second embodiment. The imaging apparatus according to the second embodiment has a pixel interpolation processing unit 802 added to the configuration described with reference to FIG. 8 in the first embodiment. Since other configurations are the same as those in FIG. 8, the same reference numerals are given and description thereof will be omitted as appropriate.

Hereinafter, as described with reference to FIG. 10(b), the process when the image signal for phase difference detection and the image signal for dynamic range expansion are alternately read out from the imaging device 400 for each row will be described.

A high-gain A signal read out at a high gain, an A+B signal read out at a high gain, and an A+B signal read out at a low gain output from the image sensor 400 are inputted to a distributor 401 . A distributor 401 separates and outputs a signal used for image (low gain A+B signal, high gain A+B signal) and a signal used for phase difference detection (high gain A signal, high gain A+B signal).

B signal generation section 408 subtracts high gain A from the high gain A+B signal output from distributor 401 to generate a high gain B signal. The phase difference detection section 403 detects a phase difference from the phase difference detection signals (high gain A signal and high gain B signal) output from the B signal generation section 408 . A focus calculation unit 404 performs focus calculation based on the phase difference information detected here and the focus position of the lens. Based on the obtained focus information, the imaging apparatus notifies the user of the focus information and performs focus control of the lens.

On the other hand, the pixel interpolation processing unit 802 interpolates pixel signals from upper and lower lines for lines from which low gain A+B signals are not read, as will be described later. When the high gain A+B signal is saturated, the image synthesizing unit 402 synthesizes a dynamic range expanded image from the high gain A+B signal and the low gain A+B signal using an arbitrary synthesizing method.

Here, FIG. 12 shows an image of image data in each block described in FIG. FIG. 12A shows an image signal output from the imaging element 400. FIG. As shown in FIG. 12A, the image signal output from the image sensor 400 is in a state in which the high gain A signal and the low gain A+B signal are alternately read out between the high gain A+B signals.

FIG. 12B shows image signals separated by the distributor 401 and input to the pixel interpolation processing unit 802 . The high gain A signal used for phase difference detection is thinned out by the distributor 401 because it is not necessary to expand the dynamic range. There is no low gain A+B signal for .

FIG. 12(c) is a diagram showing an image signal output from the pixel interpolation processing unit 802. As shown in FIG. The line from which the high gain A signal is read shows how the low gain A+B signal is interpolated using adjacent upper and lower low gain A+B signals.

Finally, FIG. 12(d) is a diagram showing the image signal input to the B signal generator 408. As shown in FIG. Since a signal for expanding the dynamic range is not required for the phase difference detection process, the low gain A+B signal and the high gain A+B signal, which are image signals for expanding the dynamic range of lines from which the high gain A signal is not read out, are distributed to the distributor. 401 thins out and outputs.

As described above, according to the second embodiment, it is possible to expand the dynamic range and detect the phase difference in the entire pixel region even when a smaller number of image signals are read.

<Third Embodiment>
Next, a third embodiment of the invention will be described. Note that the image pickup device in the third embodiment is also the same as that described in the first embodiment, so the description is omitted here.

In the above-described second embodiment, in the line from which the high gain A signal is read out for phase difference detection, interpolation is performed using adjacent upper and lower low gain A+B signals to generate an image signal for dynamic enlargement. Ta. However, if the image signal is interpolated from above and below, the vertical resolution will be lowered. Therefore, in this embodiment, the brightness level of the image signal for phase difference detection is detected, and if the brightness level is equal to or less than a predetermined value and the phase difference is equal to or less than a predetermined value, the interpolated low gain A+B signal is not used, but the position signal is detected. The image signal for phase difference detection is used as the image signal for dynamic range expansion. As a result, control is performed so as not to lower the vertical resolution.

FIG. 13 is a block diagram showing a schematic configuration of an imaging device according to the third embodiment. The imaging apparatus according to the present embodiment has the configuration described in the second embodiment with reference to FIG. Also, the processing of distributor 401 is different from that shown in FIG. The rest of the configuration is the same as the configuration described with reference to FIG. 8 in the first embodiment and the configuration described with reference to FIG. 11 in the second embodiment. Description is omitted as appropriate.

In the third embodiment, the distributor 401 outputs three signals to be used for images: a high gain A signal, a low gain A+B signal, and a high gain A+B signal. These signals output from distributor 401 are input to luminance detection section 902 . The luminance detection unit 902 detects the luminance of the high gain A signal and outputs the detection result to the line selection processing unit 903 . The line selection processing unit 903 selects whether or not to use the high gain A signal as the image signal for expanding the dynamic range based on information from the luminance detection unit 902 and the phase difference detection unit 403 .

FIG. 14 is a flowchart showing processing in the third embodiment. First, in S300, the luminance detection unit 902 detects the luminance level of the high gain A signal of the input line. Next, in S301, the phase difference detection unit 403 detects the phase difference between the high gain A signal and the high gain B signal.

In S302, the line selection processing unit 903 determines whether the luminance level of the high gain A signal detected in S300 is equal to or less than a predetermined value Th1 (threshold value or less). If it is equal to or less than the predetermined value Th1, the process proceeds to S303, and if it is greater than the predetermined value Th1, the process proceeds to S305.

In S303, based on the detection result in S301, it is determined whether the phase difference between the high gain A signal and the high gain B signal is equal to or less than a predetermined value Th2. If it is equal to or less than the predetermined value Th2, it is determined that the focus state is close to the in-focus state, and the process proceeds to S304. If it is greater than the predetermined value Th2, the process proceeds to S305.

In S304, the high gain A signal is selected as the image signal for expanding the dynamic range. On the other hand, in S305, the use of the low-gain A+B signal interpolated from above and below is selected as the image signal for expanding the dynamic range, and the pixel interpolation processing unit 802 generates up-and-down interpolation data in S306.

In S307, the image synthesizing unit 402 performs dynamic range expansion processing using the high gain A signal or the low gain A+B signal.

As described above, according to the third embodiment, by using the high gain A signal when the luminance level of the high gain A signal is equal to or less than the predetermined luminance and in the in-focus state or close to the in-focus state, The dynamic range can be expanded without sacrificing vertical resolution.

<Fourth Embodiment>
Next, a fourth embodiment of the invention will be described. Note that the imaging device in the fourth embodiment is also the same as that described in the first embodiment, so the description is omitted here.

In the fourth embodiment, in addition to the third embodiment, the amount of motion of the subject is detected during video shooting, and if the amount of motion is less than a predetermined amount, the drive of the image sensor is changed, readout for phase difference detection and dynamic detection are performed. A process of replacing readout for range expansion is performed for each frame. Also, when synthesizing images, the image signals of the preceding and succeeding frames are used to prevent deterioration of the vertical resolution.

FIG. 15 is a block diagram showing a schematic configuration of an imaging device according to the fourth embodiment. The configuration shown in FIG. 15 is obtained by adding a motion vector detection unit 909 and a memory 910 to the configuration described with reference to FIG. 13 in the third embodiment. Since other configurations are the same as those in FIG. 13, the same reference numerals are given and description thereof will be omitted as appropriate.

The motion vector detection unit 909 detects the amount of motion of the subject and outputs the detection result to the image composition unit 402 and the control unit 405 . The memory 910 can temporarily store the image signal, and image synthesis can be performed using the image signals of the preceding and succeeding frames.

FIG. 16 is a flow chart showing processing in the fourth embodiment. The same step numbers are given to the same processes as those described with reference to FIG. 14 of the third embodiment, and the description thereof will be omitted as appropriate.

In S304, when the high gain A signal is selected as the image signal for dynamic range expansion, in S407, the image synthesizing unit 402 performs dynamic range expansion using the high gain A signal as described in the third embodiment. process.

On the other hand, if the high-gain A signal is not selected, the process proceeds from S303 to S408, and in S408, it is determined whether or not the amount of motion of the subject is equal to or greater than a predetermined value Th3. If it is equal to or greater than the predetermined value Th3, the process advances to S305 to select upper and lower interpolated data interpolated from the low gain A+B signals of the upper and lower pixels. In S306, the pixel interpolation processing unit 802 generates upper and lower interpolation data, and in S410 , the image synthesizing unit 402 performs dynamic range expansion processing using the low gain A+B signal.

On the other hand, if the amount of motion of the subject is less than the predetermined value Th3, driving of the image sensor 400 is changed in S409. Here, for each frame, the case of reading the high gain A signal and the high gain A+B signal (the reading method of FIG. 6B), and the case of reading the high gain A+B signal and the low gain A+B signal (the reading method of FIG. 7A) method) alternately. Then, in S411, a signal interpolated from the low-gain A+B signals of the preceding and succeeding frames held in the memory 910 is used to perform dynamic range expansion processing.

As described above, according to the fourth embodiment, when the movement of the subject is small, dynamic range expansion processing is performed using signals interpolated from low-gain A+B signals of the preceding and succeeding frames, thereby suppressing deterioration in vertical resolution. be able to.

<Fifth Embodiment>
Next, a fifth embodiment of the invention will be described. Note that the image sensor in the fifth embodiment is the same as that described in the first embodiment, so description thereof will be omitted here.

FIG. 17 is a diagram showing the details from the unit pixel 101 to the AD circuit group 104 of the image sensor in which vertical output lines and column amplifiers are connected to the photoelectric conversion units A and B, respectively. Signals of the photoelectric conversion element 1101 corresponding to the photoelectric conversion unit A and the photoelectric conversion element 1102 corresponding to the photoelectric conversion unit B in FIG. is a possible configuration.

In addition, in FIG. 17, the same reference numerals are given to the same configurations as in FIG. 2, and the description thereof will be omitted. In the configuration shown in FIG. 17, in addition to the configuration shown in FIG. 2, each pixel 101 includes a charge holding unit 1108 and a pixel amplifier 1114 for independently reading partial signals from the photoelectric conversion element 1102, and also has a vertical output. It has a current control 1116 for controlling the current in line 1115 .

Also, the column amplifier 103i includes two amplifiers for amplifying the signals output from the photoelectric conversion elements 1101 and 1102 to the vertical output lines 1113 and 1115, respectively, and outputs the signals to the AD circuit group 104 in the subsequent stage. . Each AD circuit 104i includes, in addition to the configuration shown in FIG. It has a memory 1119 and a memory 1120 for

With the above configuration, the partial signals of the photoelectric conversion elements 1101 and 1102 can be read out in parallel, processed, and output. Therefore, although the circuit size increases, the readout time can be shortened.

Further, by adding the A signal and the B signal read out in the signal processing unit 106, the A+B signal can be obtained. However, in that case, the B signal generator 408 shown in FIGS.

FIG. 18 is a timing chart showing control of the column amplifier 103i when partial signals are read in parallel from the photoelectric conversion elements 1101 and 1102 with high gain and low gain in the configuration shown in FIG.

First, between times t51 and t54, the switch 200 is turned on and the switches 201 and 204 are turned off, thereby setting the gain of the column amplifier 103i to a high gain. In this state, the transfer switches 1103 and 1104 are turned on at time t52 to read out the A signal and the B signal, respectively. At this time, during the period from time t52 to time t54, the A signal and the B signal read at high gain are A/D converted in the AD circuit 104i.

Next, between times t54 and t55, the switch 200 is turned off and the switches 201 and 204 are turned on to set the gain of the column amplifier 103i to a low gain. At this time, during the period from time t54 to time t55, the A signal and the B signal read at the low gain are A/D converted in the AD circuit 104i.

From the read high gain A signal, high gain B signal, low gain A signal, and low gain B signal, the high gain A signal and high gain B signal are added in the signal processing unit 106 of FIG. is generated and output. By adding the low gain A signal and the low gain B signal in the signal processing unit 106, a low gain A+B signal is generated and output.

Although the timing chart shown in FIG. 18 describes the case where signals are read out with high gain and low gain, the present invention is not limited to this. For example, when a high gain signal is not required or when a low gain signal is not required, the reading speed can be increased by reading the signal with the required gain.

As described above, according to the fifth embodiment, an image signal used for dynamic range expansion and an image signal used for phase difference detection can be obtained in each frame without lowering the frame rate.

Although the present invention has been described in detail based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms without departing from the gist of the present invention can be applied to the present invention. included. Some of the above-described embodiments may be combined as appropriate.

For example, even when the low-gain A+B signal is interpolated using the image signals of the upper and lower pixels, the interpolation ratio of the upper and lower pixels may be arbitrarily changed according to the situation.

<Other embodiments>
The present invention may be applied to a system composed of a plurality of devices (eg, host computer, interface device, scanner, video camera, etc.) or to an apparatus composed of a single device.

Further, the present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device executes the program. It can also be realized by a process of reading and executing. It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

100: pixel area, 101: unit pixel, 103: column amplifier group, 103i: column amplifier, 111: microlens, 1101, 1102: photoelectric conversion element, 400: imaging element, 401: distributor, 402: image synthesizing unit, 403: phase difference detection unit, 404: focus calculation unit, 405: control unit, 408: B signal generation unit, 802: pixel interpolation processing unit, 902: luminance detection unit, 903: line selection processing unit, 909: motion vector detection Part 910: Memory

Claims (8)

a pixel region having a plurality of microlenses arranged in a matrix and a plurality of photoelectric conversion units configured for each microlens; an amplifying means that can be amplified using different gains; a partial signal that is a partial signal of the plurality of photoelectric conversion units; and a sum signal obtained by adding the signals of the plurality of photoelectric conversion units. an imaging device having scanning means capable of scanning an area;
a control means for controlling the imaging device;
processing means for expanding a dynamic range using the sum signal amplified using the plurality of different gains;
focus detection means for performing phase-difference focus detection using the partial signal and the added signal;
The control means is
A first mode in which a dynamic range is expanded by the processing means, and phase-difference focus detection is performed by the focus detection means using the partial signals and the sum signal amplified using a plurality of different gains. is set, controlling the amplifying means to amplify the partial signal and the sum signal using the plurality of different gains;
A phase-difference type focus detection is performed by the focus detection means without expanding the dynamic range by the processing means and using the partial signal and the sum signal amplified by using the first gain. controlling the amplifying means to amplify the partial signal and the sum signal using the first gain when mode 2 is set;
The partial signal is not read from the pixel area when a third mode is set in which the dynamic range is expanded by the processing means and the focus detection by the phase difference method is not performed by the focus detection means. controlling the scanning means and controlling the amplifying means to amplify the sum signal using the plurality of different gains;
A fourth method for expanding a dynamic range by said processing means and performing phase-difference focus detection by said focus detection means using said partial signal and said added signal amplified using said first gain. and controlling the amplifying means to amplify the partial signal using the first gain and amplify the sum signal using the plurality of different gains when the mode is set. imaging device. In the fourth mode, the control means
controlling the scanning means so that a first row for reading out the addition signal and the partial signal and a second row for reading out the addition signal alternate from the pixel area;
The first gain is used to amplify the summed signal and the partial signals read from the first row, and the plurality of different gains are used to amplify the summed signal read from the second row. 2. The imaging apparatus according to claim 1, wherein said amplifying means is controlled at a constant time. For each of the first rows, interpolating the first row using the summed signal read from the adjacent second row and amplified using a gain other than the first gain. further comprising means;
The processing means uses the sum signal read from the first row and amplified using the first gain and the signal interpolated by the interpolation means to perform 3. The imaging apparatus according to claim 2, wherein the dynamic range is expanded. The focus state detected by the focus detection means, wherein the luminance level of the partial signal read from the first row and amplified using the first gain is equal to or lower than a predetermined threshold value, and is closer to an in-focus state than a predetermined focus state, the processing means combines the summed signal read from the first row and amplified using the first gain and the portion 4. The imaging device of claim 3, wherein the dynamic range of the first row is expanded using a signal. further comprising motion detection means for detecting motion of the subject;
When the motion of the subject is equal to or less than a predetermined threshold, the control means
controlling the scanning means to read out the addition signal and the partial signal from the pixel area, and controlling the amplifying means to amplify the addition signal and the partial signal using the first gain; a first frame;
a second frame for controlling the scanning means to read out the sum signal from the pixel area and controlling the amplifying means to amplify the sum signal using the plurality of different gains;
alternately
wherein the interpolating means interpolates the first frame using the added signal amplified by a gain other than the first gain in the second frame adjacent to the first frame;
The processing means expands the dynamic range of the first frame using the added signal of the first frame amplified using the first gain and the signal interpolated by the interpolating means. 5. The imaging apparatus according to claim 3, wherein: a pixel region having a plurality of microlenses arranged in a matrix and a plurality of photoelectric conversion units configured for each microlens; an amplifying means that can be amplified using different gains; a partial signal that is a partial signal of the plurality of photoelectric conversion units; and a sum signal obtained by adding the signals of the plurality of photoelectric conversion units. A control method for an imaging device having an imaging element having scanning means capable of scanning an area,
A control step in which the control means controls the imaging device;
a dynamic range expansion step in which processing means expands a dynamic range using the sum signal amplified using the plurality of different gains;
a focus detection step in which focus detection means performs phase-difference focus detection using the partial signal and the added signal;
In the control step,
A first mode in which a dynamic range is expanded by the processing means, and phase-difference focus detection is performed by the focus detection means using the partial signals and the sum signal amplified using a plurality of different gains. is set, controlling the amplifying means to amplify the partial signal and the sum signal using the plurality of different gains;
A phase-difference type focus detection is performed by the focus detection means without expanding the dynamic range by the processing means and using the partial signal and the sum signal amplified by using the first gain. controlling the amplifying means to amplify the partial signal and the sum signal using the first gain when mode 2 is set;
The partial signal is not read out from the pixel area when a third mode is set in which the processing means expands the dynamic range and the focus detection means does not perform phase-difference focus detection. controlling the scanning means and controlling the amplifying means to amplify the sum signal using the plurality of different gains;
A fourth method for expanding a dynamic range by said processing means and performing phase-difference type focus detection by said focus detection means using said partial signal and said added signal amplified using said first gain. and controlling the amplifying means to amplify the partial signal using the first gain and amplify the sum signal using the plurality of different gains when the mode is set. control method. A program for causing a computer to execute each step of the control method according to claim 6 . A computer-readable storage medium storing the program according to claim 7 .

Images