JP6991028B2 - Imaging device and its control method, program, storage medium - Google Patents

Description

The present invention relates to a technique for driving an image pickup device in an image pickup apparatus.

Currently, in an image pickup device such as a digital camera, various arithmetic processes are performed based on the information obtained from the image pickup element. In particular, a technique for performing autofocus (AF) using the phase difference information of a subject obtained from an image sensor is well known. At that time, various methods have been proposed for reading out the image sensor in order to realize both reading that enables highly accurate focus detection and reading at low power consumption and high speed.

In Patent Document 1, a plurality of photoelectric conversion units are arranged in a single pixel, and signals of the plurality of photoelectric conversion units are read out so that signals of the plurality of photoelectric conversion units can be independently acquired by some pixels in all the pixels, and the remaining pixels are single. A method has been proposed in which the signals of the photoelectric conversion unit of the pixel are combined and then read out. By reading in this way, the phase difference information that takes a long time to be acquired is read only in the necessary area, and it is possible to shorten the operation time and reduce the power consumption.

Patent Document 2 discloses that the image pickup device is divided into a plurality of regions, and a reading method for acquiring phase difference information is performed in the first region where focus detection needs to be performed. Then, in the second region, in order to reduce power consumption, the thinning rate is larger than the read in the first region. By performing such control, power consumption can be suppressed.

Patent Document 3 discloses a method of switching a reading method based on the spatial frequency of a subject. By performing such control, precise focusing can be performed on a high-frequency subject, and power consumption can be suppressed when shooting a low-frequency subject.

Japanese Unexamined Patent Publication No. 2013-21183 JP-A-2015-79162. Japanese Unexamined Patent Publication No. 2015-165280

Currently, the phase difference information of a subject obtained from an image sensor is considered to be used in various situations. As an example, in addition to the use for AF calculation, the use for creating a defocus map (hereinafter referred to as DEF-MAP) which is the distribution of the depth information of the subject on the entire screen from the phase difference information of the entire screen. Can be mentioned. As described above, when the phase difference information is used for a plurality of different purposes (here, for AF and for DEF-MAP), the properties of the required phase difference information may differ.

For example, in the creation of DEF-MAP, the phase difference information of the entire screen is required, whereas in the acquisition of AF information, it is sufficient that the phase difference information of only the portion of interest can be acquired. Further, when classifying subjects as an application of DEF-MAP, the resolution of the phase difference information is not so required, whereas in the acquisition of AF information, it is required to read the phase difference information at a high resolution. ..

In the above-mentioned conventional technique, when the phase difference information is read from a part of the pixel group in the whole pixel, it cannot be read so as to satisfy the above-mentioned plurality of properties at the same time in one frame. Further, when the phase difference information is read from the entire pixel, the above-mentioned plurality of functions can be used, but a decrease in the frame rate cannot be avoided.

The present invention has been made in view of the above-mentioned problems, and an object thereof is to suppress a decrease in frame rate while reading out pixel signals suitable for a plurality of purposes when acquiring phase difference information. It is to provide an image pickup apparatus capable of.

The image pickup apparatus according to the present invention is an image pickup element having a pixel region in which a plurality of unit pixels having one microlens and a plurality of photoelectric conversion units arranged corresponding to the one microlens are arranged in a matrix. And, for each pixel of the first region and the second region of the pixel region, the signals of the plurality of photoelectric conversion units are grouped into a first number and the signals are read out in the first mode. Whether to use the first pixel or to use the second pixel for reading the signal in the second mode in which the plurality of photoelectric conversion units are grouped into a second number larger than the first number and the signal is read out. The first area and the second area pixels are set to the first pixel or the second pixel in row units of the matrix-like arrangement, and the first pixel and the second area are set to the first pixel or the second pixel. The row period set in the second pixel is different between the first region and the second region .
Further, the image pickup device according to the present invention has a pixel region in which a plurality of unit pixels having one microlens and a plurality of photoelectric conversion units arranged corresponding to the one microlens are arranged in a matrix. For each pixel of the image pickup element and the first region and the second region of the pixel region, the signals of the plurality of photoelectric conversion units are grouped into a first number and the signals are read out in the first mode. Is used as the first pixel to read the signal, or the plurality of photoelectric conversion units are grouped into a second number larger than the first number and the signal is read out with the second pixel. It is characterized in that a defocus map is created based on the signal in the first region, and autofocus is performed based on the signal in the second region.

According to the present invention, it is possible to provide an image pickup apparatus capable of suppressing a decrease in frame rate while reading out pixel signals suitable for a plurality of purposes when acquiring phase difference information. ..

The block diagram which shows the structure of the image pickup apparatus which concerns on 1st Embodiment of this invention. Pixel arrangement diagram in the image sensor and circuit block diagram. A circuit diagram of a unit pixel and a circuit diagram of one row of a readout circuit. Circuit diagram of AD conversion circuit. A timing chart showing an operation example of the first drive mode of the image sensor. A timing chart showing an operation example of the second drive mode of the image sensor. The figure which shows the example of division of the area of an image sensor, and the example of combination of a drive mode. The figure which shows the example of the time required for the whole system required for the operation of 1 frame in 2nd Embodiment. The circuit block diagram of the image pickup element in the 2nd Embodiment. The figure which shows the example of shortening the time required for the whole system required for the operation of 1 frame in 2nd Embodiment. The figure which shows the read setting example in 3rd Embodiment. The figure which shows the image processing example and the readout setting example in 4th Embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

<First Embodiment>
FIG. 1 is a block diagram showing a configuration of an image pickup apparatus according to a first embodiment of the present invention. In FIG. 1, the first lens 100 is arranged at the tip of the photographing optical system 120. The aperture 101 adjusts the amount of light at the time of shooting by adjusting the aperture diameter thereof. The second lens 102 and the third lens 103 are driven by the focus actuator 116 described later, and move back and forth in the optical axis direction to adjust the focus of the photographing optical system 120.

The focal plane shutter 104 adjusts the exposure seconds when shooting a still image. The optical low-pass filter 105 is used to reduce false colors and moire in captured images. The image pickup device 106 photoelectrically converts the optical image of the subject formed by the photographing optical system 120 into an electric signal. The image sensor 106 is controlled by the CPU 109 described later.

The DSP (Digital Signal Processor) 107 performs image processing. The DSP 107 performs processing such as correction and compression of the image data captured by the image sensor 106. Further, it has a function of separating A image data and B image data, which will be described later, and a correlation calculation function of performing a correlation calculation using an image signal. The RAM 108 has a function of a signal holding means for holding the output data from the image pickup element 106, a function of an image data storing means for storing the image data processed by the DSP 107, and a work memory when the CPU 109 described later operates. Combines functions. In the present embodiment, these functions are realized by using the RAM 108, but other types of memory may be used as long as the access speed is sufficiently high and there is no problem in operation. It is possible. Further, in the present embodiment, the RAM 108 is arranged outside the DSP 107 and the CPU 109, but a part or all of the functions thereof may be built in the DSP 107 and the CPU 109.

The CPU 109 comprehensively controls the operation of the image pickup apparatus. The CPU 109 executes a program for controlling each part of the image pickup apparatus. It also has a function of controlling the focus drive circuit 115 described later and adjusting the focus of the photographing optical system 120 by using the result of the correlation calculation output from the DSP 107. The display unit 110 displays captured still images, moving images, menus, and the like. The operation unit 111 sets shooting commands, shooting conditions, and the like to the CPU 109. The recording medium 112 is a detachable recording medium for recording still image data and moving image data. The ROM 113 stores a program loaded and executed by the CPU 109 in order to control the operation of each part.

The shutter drive circuit 114 drives and controls the focal plane shutter 104. The focus drive circuit 115 is a focus position changing means for changing the focus position of the photographing optical system 120, controls the focus actuator 116 based on the output of the CPU 109, and moves the second lens 102 and the third lens 103 in the optical axis direction. The focus is adjusted by driving forward and backward. The aperture drive circuit 117 controls the aperture actuator 118 to control the aperture of the aperture 101.

Next, the configuration of the image pickup device 106 will be described with reference to FIG. FIG. 2A shows a pixel array (pixel region) in which the unit pixels of the image pickup device 106 are arranged in a matrix. In FIG. 2A, a plurality of unit pixels 201 are arranged in the matrix direction in the pixel array 200. Each unit pixel 201 has a microlens 202. Each unit pixel 201 has two photodiodes (PD) 203a, 203b as photoelectric conversion means under a single microlens 202. By adopting this configuration, the PD203a and 203b have a pupil division configuration, and different images of the same subject with a phase difference are incident on the two PD203a and 203b. Two separate images of the same subject having a phase difference are referred to as an A image and a B image, and PD203a constitutes an A image photoelectric conversion unit and PD203b constitutes a B image photoelectric conversion unit.

Next, the circuit configuration of the image pickup device 106 will be described with reference to FIG. 2 (b). In the pixel array 200, a plurality of unit pixels 201 (m + 1) in the horizontal direction and (n + 1) in the vertical direction are arranged. The vertical scan circuit control unit 204 can set the first read setting shown by 205a and the second read setting shown by 205b. The vertical scanning circuit 206 transfers a pixel signal by sending a pulse to the pixel. The vertical scan circuit control unit 204 controls the vertical scan circuit 206 based on the programmed read settings. The timing generator (TG) 207 supplies each circuit block with the control pulse required for reading according to the settings programmed by the CPU 109. Although the TG 207 is built in the image pickup device 106 in the present embodiment, it may be configured to be arranged outside the image pickup device 106.

The pixel signal photoelectrically converted by each unit pixel 201 is output line by line to the vertical output line 208 by the drive signal supplied from the vertical scanning circuit 206 as described above. When acquiring the pixel signal of a certain row, whether the vertical scanning circuit 206 reads the pixel signal in the first drive mode or the second drive mode, which will be described later, is controlled by the vertical scanning circuit control unit 204. The constant current source 209 is combined with the pixel amplifier transistor 303 described later to form a source follower circuit. The readout circuit 210 has a function of amplifying the output from the vertical output line 208 of each column. The AD conversion circuit (ADC) 211 converts the output of the read circuit 210 into a digital signal. The image signals converted by the ADC 211 are sequentially selected by the horizontal scanning circuit 212, subjected to the processing described later by the output unit 213, and output as a digital signal to the outside of the image pickup device 106.

FIG. 3A is a diagram illustrating a detailed circuit configuration of the unit pixel 201. In FIG. 3A, PD203a and PD203b are PDs constituting the unit pixel 201 under the same microlens described above. The transfer switch TxA (300a) for the A image is controlled by the signal φtxa, and the transfer switch TxB (300b) for the B image is controlled by the signal φtxb. By setting the signals φtxa and φtxb to High (hereinafter, High is referred to as H), the electric charge accumulated in the photoelectric conversion unit can be transferred to the floating diffusion unit (FD) 301. Since the ON / OFF of this transfer switch can be controlled independently, the signal for the A image and the signal for the B image can be transferred independently. The reset switch 302 is a switch for initializing the FD 301, and is controlled by the signal φpres. The pixel amplifier transistor 303 is connected to the constant current source 209 via the select switch 304 and the vertical output line 208. When the input signal φsel of the select switch 304 becomes H, the pixel amplifier transistor 303 is connected to the constant current source 209 to form a pixel amplifier. Since the FD301 is connected to this pixel amplifier, the charge transferred from the PD203a and PD203b to the FD301 is converted into a voltage value according to the amount of charge by this pixel amplifier and output as a pixel signal to the vertical output line 208. It will be.

The readout circuit 210 will be described in detail with reference to FIG. 3 (b). The pixel signal output to the vertical output line 208 is input to the inverting input terminal of the operational amplifier 308 through the clamp capacitance 305. A reference voltage Vref is input to the non-inverting input terminal of the operational amplifier 308. The switch 307 is a switch for short-circuiting the feedback capacitance 306, and is controlled by the signal φcfs.

The configuration of the ADC 211 and its driving will be described in detail with reference to FIG. The output of the readout circuit 210 and the lamp signal from the lamp signal generation circuit 401 are input to the comparator 400. The output level of the lamp signal generation circuit 401 increases with the passage of time when the signal φadcres input from the TG 207 via the terminal 402 becomes Low (hereinafter, Low is referred to as L). Then, when the signal level exceeds the level of the signal output from the readout circuit 210, the output of the comparator 400 is switched from L to H. The output of the comparator 400 is connected to the latch selection circuit 404. Then, when the signal φssel input from the TG 207 via the terminal 403 is L, it is input to the N signal latch 408, and when the φssel is H, it is input to the S signal latch 409. The N signal latch 408 and the S signal latch 409 are reset when the signals φnres and φsres input from the TG 207 via the terminals 405 and 406 become H, respectively, and operate as a latch when L, as follows.

First, when φssel is L, the N signal latch 408 latches the output of the counter 407 at the moment when the output of the comparator 400 is switched from L to H. This counter value is a value expressing the digital value of the output of the read circuit 210. Since this latch 408 is an N signal latch, it is controlled so that the result of digitally converting the noise signal is held in the latch 408. Next, when φssel is H, the output of the comparator 400 is connected to the latch 409, and the output of the readout circuit 210 is converted into a digital value by performing the same operation as the latch 408 described above. Since this latch 408 is an S signal latch, it is controlled so that the result of digitally converting the signal to which the optical signal and the noise signal are added is held in the latch 408. The outputs of the latches 408 and 409 are connected to the column selection circuit 410 and are output to the output unit 213 by the signal φhsr_i (i is the column number 0 to m) from the horizontal scanning circuit 212 as described below. ..

The column selection circuit 410 in column i is in the OFF state when the signal φhsr_i is L and is disconnected from the output unit 213, but is turned ON when φhsr_i is H, and the N signal latch and S in the column. The output of the signal latch is output to the output unit 213.

When the horizontal scanning circuit 212 detects the rising edge of the signal φhsr input from the TG 207 via the terminal 411 when all φhsr_i are L, the horizontal scanning circuit 212 sets φhsr_0 to H. After that, when the rising edge of φhsr is detected when φhsr_i is H, φhsr_i is set to L and φhsr_ (i + 1) is set to H. Then, when the rising edge of φhsr is detected when φhsr_m is H, all φhsr_i are set to L. In this way, the horizontal scanning circuit 212 sequentially outputs the outputs of the latches from the 0th column to the mth column to the output unit 213 to perform horizontal scanning. The output unit 213 has a function of subtracting the output of the N signal latch 408 from the output of the S signal latch 409 and then outputting the output to the outside.

Subsequently, a timing chart showing an example of reading the image signal of the image sensor 106 for each operation of the first drive mode (mode in which the phase difference cannot be detected) and the second drive mode (mode in which the phase difference can be detected) is shown. This will be described with reference to FIGS. 5 and 6.

Using the timing chart of FIG. 5, a method of acquiring a pixel signal of one row by the first drive mode will be described in detail.

Before the time t501, the signal φpres is H and the FD301 is reset. Subsequently, PD203a and 203b are reset by setting the signals φtxa and φtxb to H at time t501. By setting the signals φtxa and φtxb to L at time t502, charge accumulation of PD203a and 203b is started. The accumulation time is from this point to the time when the signal is read, and is between the time t502 and the time t511 in the figure. By setting the signal φsel to H at time t503, the pixel amplifier transistor 303 is connected to the constant current source 209, and the pixel amplifier is put into an operating state. By setting the signal φpres to L at time t504, the reset of the FD 301 is released. When the reset is released, the signal of the FD 301 is read out as a noise signal (hereinafter referred to as N signal) on the vertical output line 208 and output to the read circuit 210.

In the read circuit 210, by setting the signal φcfs to L at time t505, the output buffer state of the reference voltage Vref in the operational amplifier 308 is released, and the N signal of the read circuit 210 is output from the operational amplifier 308. At this time, the input from the read circuit 210 to the comparator 400 built in the ADC 211 starts to rise as shown by 500 in the figure. Then, by setting the signal φadcres to L at time t506, the reset of the lamp signal generation circuit 401 and the counter 407 is released, and at the same time, the reset of the latch 408 is released by setting the signal φnres to L, and the AD conversion operation is performed. Start. The lamp signal input from the lamp signal generation circuit 401 to the comparator 400 is shown by 501 in the figure, and the value of the counter 407 is shown by 502 in the figure. The ramp signal becomes larger than the input 500 from the readout circuit at time t507, at which time the output of the comparator 400 rises from L to H. Since the signal φssel becomes L at time t507, the N signal latch 408 is selected in the latch selection circuit 404, and the output of the comparator is input to the latch 408. The latch 408 latches the output of the counter 407 at time t507. Here, the value is N as shown by 503 in the figure. Thus, at time t507, the value of the latch 408 becomes N. At time t508, both the lamp signal 501 and the counter value 502 become maximum values, and at that timing, the signal φadcres is set to H and the AD conversion operation of the N signal is terminated.

At time t509, the signals φtxa and φtxb are set to H, and the optical charge converted by PD203a and 203b is transferred to the FD301. The electric charge of the signal component stored in the FD 301 is input to the comparator 400 from the readout circuit 210 as a signal of the signal component + noise component (hereinafter referred to as S + N signal) through the vertical output line 208 and the operational amplifier 308. Input 500 begins to increase at time t509. At time t510, by setting the signal φssel to H, the latch selected by the latch selection circuit 404 is switched to the S signal latch 409. After the signals φtxa and φtxb are set to L at time t511 and the transfer of optical charges is completed, the output of the readout circuit 210 is waited for stabilization until time t512. By setting the signal φadcres and the signal φsres to L at time t512, the AD conversion operation of the S + N signal is started. This S + N signal includes a pixel signal (S signal) obtained by converting the optical charge stored in the PD 203a and 203b, and a noise signal (the above-mentioned N signal) such as an offset signal due to variation in each column of the readout circuit 210. .. By subtracting the N signal from this S + N signal, the noise component included in the S + N signal can be canceled, and only the S signal which is a pixel signal component can be obtained.

The operation of the AD conversion of the S + N signal is performed in the same manner as the AD conversion of the N signal, the output of the comparator 400 rises from L to H at time t513, and the output of the latch 409 becomes the value S + N of the counter 407 at that time. Change. Here, the value is S + N as shown by 504 in the figure. Thus, at time t517, the value of the latch 409 becomes S + N. The output of this latch 409 becomes the AD conversion value of the S + N signal. At time t514, the signal φadcres becomes H, and the AD conversion operation ends.

At time t515, the rising edge of the signal φhsr is input to the horizontal scanning circuit 212. Then, from the state where all the outputs from the horizontal scanning circuit are L, only the signal φhsr_0 becomes H, the switch of the horizontal selection circuit 410 in the 0th column is connected, and the latches 408 and 409 are connected to the output unit 213. The output unit 213 subtracts the digital value of the N signal, which is the output value of the latch 408, from the digital value of the S + N signal, which is the output value of the latch 409, to obtain the digital value of only the S signal. Then, the value is output to the outside of the image sensor 106. When the digital value of the S signal is output from the 0th column in this way and then the pulse is input to the signal φhsr again, only the signal φhsr_1 becomes H, and the digital value of the S signal is output from the 1st column in the same operation. Will be done. In this way, the S signal is sequentially read in the row direction, and the signals from the 0th column to the mth column are read out. This horizontal scan ends at time t516.

At time t517, each pulse is returned to the initial state to initialize the pixel state. FD301 is initialized by setting the signal φpres to H. By setting the signal φcfs to H, the output of the operational amplifier 308 is set to the output buffer state of the reference voltage Vref. By returning the signal φssel to L, the latch selection circuit 404 selects the N signal latch 408. By setting the signals φnres and φsres to H, the outputs of the latches 408 and 409 are reset. At time t518, the signal φpsel is set to L, and the row selection state is canceled. The above is the read operation in the first drive mode.

Next, using the timing chart of FIG. 6, a method of acquiring a pixel signal of one row by the second drive mode will be described in detail. Since the operation of the timing chart of FIG. 6 has many parts similar to the operation of the timing chart of FIG. 5, the description will be focused on different parts.

From time t601 to time t608, the operation is exactly the same as the operation from time t501 to time t508 in the timing chart of FIG. By the operation up to this point, the digital value 600 of the N signal is held in the latch 408 for the N signal. At time t609, the signal φtxa is set to H, and the optical charge converted by PD203a is transferred to FD301. The signal φtxb set to H at time t509 in the timing chart of FIG. 5 is left as L here. By doing so, only the signal of the photoelectric conversion unit for A image can be transferred to the FD 301. After that, until the time t616, the horizontal transfer is completed by the same operation as the time t516 of the timing chart of FIG. At this time, the pixel signal output from the S signal latch 409 is an A image signal + a noise signal 601. Therefore, by subtracting the digital value 600 of the N signal in the output unit 213, only the signal of the PD203a, which is the photoelectric conversion unit for the A image in the unit pixel 201, can be read out to the outside of the image sensor 106 by this horizontal transfer. ..

At time t617, the signal φthres is set to H. By this operation, the S signal latch 409 is reset, and it becomes possible to perform an AD conversion operation using the S signal latch 409 again. At time t618, the signals φtxa and φtxb are set to H, and the optical charge converted by the PD 203b is transferred to the FD 301. By this operation, in FD301, the optical charge of PD203a and the optical charge of PD203b can be added. At this time, the FD 301 already holds the optical charge of the PD 203a transferred from the time t609 to the time t611. Therefore, by setting only the signal φtxb to H, the signal of PD203b may be transferred to FD301, and the optical charge of PD203a and the optical charge of PD203b may be added by FD301. In the present embodiment, in order to make the circuit state at the time of A + B image signal transfer the same as that of the first drive mode, φtxa is also operated to be H. By the above operation, all the charges of the photoelectric conversion unit under the unit pixel can be transferred to the FD 301.

The operation from time t619 to time t626 is the same as the operation from time t511 to time t519 in the timing chart of FIG. At this time, since the pixel signal output from the S signal latch 409 is an A + B image signal + N signal 602, by subtracting the digital value 600 of the N signal in the output unit 213, all the photoelectric conversion units in the unit pixel The signal can be read out to the outside of the image pickup device 106. The above is the read operation by the second drive mode.

As described above, an example of the first drive mode is shown using the timing chart of FIG. 5, and an example of the second drive mode is shown using the timing chart shown in FIG. According to the above example, in the first drive mode, a signal (referred to as an A + B image signal) obtained by synthesizing the outputs of the two PD203a and 203b under the unit pixel is read out. On the other hand, in the second drive mode, of the two PDs under the unit pixel, the A image signal is first read, and then the A + B image signal is read. For example, in the DSP 108, the B image signal can be separated by subtracting the A image from the A + B image obtained by the second drive mode. Then, by detecting the phase difference between the A image signal and the B image signal, the defocus amount of the photographing optical system 120 can be detected.

As described above, in the first drive mode, the signal of the photoelectric conversion unit under the unit pixel is synthesized and read out, whereas in the second drive mode, the signal of the photoelectric conversion unit under the unit pixel is synthesized and read out. Can be read independently. Therefore, in the second drive mode, more signals of unit pixels are divided into pupils and read out than in the first drive mode. Here, if the reading of the unit pixel of the first drive mode is divided into one (no division), the second drive mode is the reading in which the photoelectric conversion unit is divided into two groups.

In this way, when the signals of the plurality of photoelectric conversion units under the unit pixel are divided and read in the second drive mode, the phase difference information can be obtained, but compared with the reading in the first drive mode. And the read time is extended. Therefore, in order to secure the frame rate of the image, it is necessary to minimize the number of read lines in the second drive mode. Therefore, in the present embodiment, the screen is divided into a first region and a second region according to the purpose of use of the phase difference information, and the image signal is acquired separately in each region by the second drive mode. Set the group.

FIG. 7A shows an example of dividing the screen into a first area and a second area (divided area). The area 700 shows the entire screen, for example, the entire pixel array 200. Reference numeral 701 is a focus detection frame (focus detection area), and the DSP 107 performs a correlation calculation based on the subject information in the focus detection frame 701. As described above, since it is desired to perform focus detection with high accuracy in the region including this focus detection frame, it is necessary to read out the phase difference information with high resolution. Further, in the region not including the focus detection frame 701, as described above, since the defocus map (DEF-MAP) is created, the entire phase difference information may be read out at a low resolution. Therefore, both the area indicated by dots and the area indicated by diagonal lines are designated as the first area 702, and only the area indicated by the diagonal lines is designated as the second area 703. Further, in the region 700, a region that is not selected by either the first region 702 or the second region 703 is defined as a third region. The area setting means for setting this area can be realized by the CPU 109, for example, by calculating and setting the area by the CPU 109. As in the present embodiment, the first region and the second region may or may not overlap.

FIG. 7B shows an example in which the first region 702 is read by the first read setting 205a and the second region 703 is read by the second read setting 205b. The first read setting 205a is a read setting for selecting the start line of the region and the line to be read from the line in the second drive mode (mode in which the phase difference can be detected) every constant first line cycle 704. It has become. The row selected by the first readout setting 205a is the row indicated by the alternate long and short dash line in the figure. The second read setting 205b is a read setting that selects the start line of the region and the line to be read from the line in the second drive mode (mode in which the phase difference can be detected) every fixed second line cycle 705. It has become. The row selected by the second read setting 205b is the row shown by the solid line in the figure. The ratio of the number of rows read in the second drive mode to the number of rows read in the first drive mode is different between the first read setting 205a and the second read setting 205b.

In the present embodiment, the second region 703 is a region included in the first region 702. Therefore, in the second area 703, the line not selected by the second read setting 205b may be the line selected by the first read setting 205a.

Rows not selected for either the first read setting 205a or the second read setting 205b are read in the first drive mode (mode in which the phase difference cannot be detected).

The first read setting 205a and the second read setting 205b determined based on the first area 702 and the second area 703 are set by the vertical scanning circuit control unit 204. The vertical scanning circuit control unit 204 controls the vertical scanning circuit 206, and switches whether the acquisition of a certain row is performed by the first drive mode or the second drive mode.

As described above, according to the first embodiment, while acquiring the phase difference information at the optimum resolution in each region, the acquisition of pixels by the second drive mode, which takes a long time to drive, is minimized. be able to. This makes it possible to acquire phase difference information at the resolution required for each of the correlation calculation for AF and the creation of the DEF-MAP while suppressing the decrease in the frame rate.

In the present embodiment, as the second drive mode, a method of acquiring an A image and then acquiring an A + B image is shown in the timing chart of FIG. However, the present invention is not limited to this, and for example, the FD 301 may be reset after acquiring the A image signal, and then the B image signal may be read out. In this case, it is necessary to add the digital value of the A image signal and the digital value of the B image signal in order to generate the A + B image signal for the image.

Further, in the present embodiment, the case where an image pickup device having a structure in which the photoelectric conversion unit under a unit pixel is divided into two is used has been described, but the present invention is not limited thereto. For example, in an image sensor having a structure in which the photoelectric conversion unit under a unit pixel is divided into four vertically and horizontally, more drive modes can be considered. For example, of the four photoelectric conversion units under a unit pixel, the entire screen can be obtained by combining a drive mode in which the vertical direction is combined and the horizontal direction is not combined, and a drive mode in which the signals of all the photoelectric conversion units are read independently. It is possible to read out the pixels of.

By the way, the second drive mode is a drive mode characterized in that a larger number of unit pixels are divided into pupils and read out than in the first drive mode. Therefore, when the entire screen is read out by the drive mode as described above for an image sensor having a structure in which the photoelectric conversion unit under the unit pixel is divided into four, the vertical direction is combined among the four photoelectric conversion units under the unit pixel. However, the drive mode for reading in the horizontal direction without synthesizing is set as the first drive mode. Further, the drive mode in which the signals of all the photoelectric conversion units are independently read out is set as the second drive mode. At this time, it can be considered that the reading of the unit pixel of the first drive mode is divided into two and the reading of the second drive mode is divided into four.

As described above, by configuring the photoelectric conversion unit under the unit pixel into four parts in the vertical and horizontal directions, it is possible to obtain not only the horizontal phase difference information of the subject but also the vertical phase difference information. become able to. By applying the present invention even in such a case, it is possible to read out a signal suitable for a plurality of purposes while suppressing a decrease in the frame rate.

Further, in the present embodiment, it has been described that the pixel group read by the second drive mode is selected in line units according to the read setting, but the present invention is not limited to this.

(Second embodiment)
In the first embodiment, a method of reducing a decrease in the frame rate with respect to reading pixel information has been described. However, considering the time for performing focus drive by performing correlation calculation based on pixel information or the time for generating DEF-MAP and performing processing calculation based on DEF-MAP, the frame rate of the entire system decreases. May not be fully suppressed.

FIG. 8 shows an operation example of one frame of the entire system when acquiring a still image. FIG. 8A is an example in which the correlation calculation and the lens driving time are long. In this case, since it takes a long time to finish driving the lens, the frame rate of the entire system is lowered. FIG. 8B is an example in which the time required for DEF-MAP generation and processing calculation is long. In this case, since it takes a long time to complete the processing operation, the frame rate of the entire system is lowered. In order to deal with such a decrease in the frame rate of the entire system, it is desirable to first acquire the information used for the calculation.

Further, when acquiring a moving image using the slit rolling shutter, it is necessary to correctly correspond the temporal order of row reading with the positional order in the image of one frame. If such an association is not made, the temporal relationship is switched between the top and bottom of the image, resulting in an unnatural image. In this case, as shown in FIG. 8, even if the frame rate of the entire system is lowered due to a certain calculation, the information used for the calculation cannot be acquired first.

The calculation that causes the frame rate to drop differs depending on the shooting mode. In addition, whether or not the slit rolling reading is performed also differs depending on the shooting mode. Therefore, in the present embodiment, a method of changing at least the acquisition order of the pixel signal of the opening (the signal of a pixel that is not a light-shielded pixel such as a light-shielding pixel) is shown depending on the shooting mode.

It should be noted that there is known a method of increasing the number of pixels by using pixels other than openings, such as light-shielding pixels, for various corrections. If it is necessary to read out the pixels other than the opening first in order to use it for such correction, the pixels other than the opening may be read first regardless of the acquisition order of the pixel signals in the opening.

FIG. 9 shows the configuration of the image pickup device 106 in this embodiment. The configuration shown in FIG. 9 differs from the configuration of the first embodiment shown in FIG. 2B only in that it has an image acquisition method setting unit 900. The image acquisition method setting unit 900 controls the operation of the vertical scanning circuit 206 according to the set shooting mode. More specifically, the vertical scanning circuit 206 is controlled so that an image is acquired as described below.

When a moving image is acquired by a slit rolling shutter, the reading order cannot be changed as described above. At this time, the image acquisition method setting unit 900 controls the vertical scanning circuit 206 to read sequentially in the order of arrangement regardless of the setting of the reading method setting unit 204. By doing so, the temporal relationship between the top and bottom of the image can be maintained regardless of the setting of the reading method, and the discomfort of the image can be suppressed.

When a still image is acquired and the time for correlation calculation and lens driving is long as shown in FIG. 8A, the pixel information used for the correlation calculation is acquired in advance, and the pixel information of all pixels is acquired. I want to start the operation before it is completed. At this time, the image acquisition method setting unit 900 controls the vertical scanning circuit 206 so as to perform reading according to the following procedure.

First, the row selected by the second read setting 205b set in the vertical scanning circuit control unit 204 is first acquired by the second drive mode. Then, after all the acquisitions of such rows are completed, the other rows are read. By doing so, as shown in FIG. 10A, the correlation operation can be started prior to the completion of reading the image, and as a result, the decrease in the frame rate of the entire system can be suppressed.

When a still image is acquired and the time for DEF-MAP generation and processing calculation is long as shown in FIG. 8B, the pixel information used for DEF-MAP generation is acquired in advance, and the image of all pixels is acquired. I want to start the calculation before the information acquisition is completed. At this time, the image acquisition method setting unit 900 controls the vertical scanning circuit 206 so as to perform reading according to the following procedure.

First, the row selected by the first read setting 205a set in the vertical scanning circuit control unit 204 is first acquired by the second drive mode. Then, after all the acquisitions of such rows are completed, the other rows are read. By doing so, as shown in FIG. 10B, DEF-MAP generation can be started prior to the completion of image reading, and as a result, a decrease in the frame rate of the entire system can be suppressed.

As described above, according to the second embodiment, by providing the image acquisition method setting unit 900, it is possible to change at least the acquisition order of the pixel signals of the openings according to the set shooting mode. As a result, it is possible to suppress a decrease in the frame rate of the entire system.

In the present embodiment, a method of acquiring a moving image by a slit rolling shutter has been described, but the present invention is not limited thereto. For example, it is also possible to acquire a moving image with an image sensor having a global electronic shutter function. In that case, the temporal order of reading and the positional order in the image can be acquired independently. Therefore, even at the time of acquiring a moving image, as described in the present embodiment, it is possible to suppress a decrease in the frame rate of the entire system by first reading out the information used for the time-consuming calculation.

(Third embodiment)
As shown in FIG. 7 of the first embodiment, the first region and the second region can be set overlapping. In this case, the overlapping region is read by the second drive mode in both the pixel group selected by the first read setting 205a and the pixel group selected by the second read setting 205b. In the read setting shown in FIG. 7B, the pixel group selected by the first read setting 205a and the pixel group selected by the second read setting 205b do not overlap, but the read setting is set. Some may overlap.

FIG. 11 simply shows a state in which the pixel groups selected by the read setting overlap. FIG. 11 is an extraction of the nth to n + 10th rows of the region 703 in which the first region and the second region of FIG. 7A overlap. The shaded area shown by 1100 in the figure is the line selected by the first read setting 205a, and the shaded area shown by 1101 in the figure is the line selected by the second read setting 205b. .. The nth row is selected by the first read setting 205a. When the first read setting 205a selects a row to be read in the second drive mode with a row cycle of 10 rows, the n + 10th row is also selected by the first read setting 205a. The n + 1th line is selected by the second read setting 205b. When the second read setting 205b selects a row to be read in the second drive mode with a row cycle of three rows, the n + 10th row is also selected by the second read setting 205b. In this case, the n + 10th line is selected by both the first read setting 205a and the second read setting 205b.

In the reading of the pixel signal as described in the first embodiment, the reading from PD203a and 203b is a destructive reading. When the same pixel is read out a plurality of times in such an image sensor, an incorrect value will be acquired from the second time onward, and incorrect data will be used for the calculation using the phase difference information.

Therefore, when the same pixel group is selected for both the first read setting 205a and the second read setting 205b as in the present embodiment, the reading is performed only once. Then, the overlappingly selected pixel groups (pixels belonging to the n + 10th row in the present embodiment) are calculated by the calculation using the pixel information selected by the first read setting 205a and by the second read setting 205b. It is used in common for both operations using the selected pixel information.

As described above, according to the third embodiment, even when the pixels to be acquired in the second drive mode are duplicatedly selected in the plurality of read settings, the operation is such that the abnormal output is not used in the calculation. It is possible to make it.

(Fourth Embodiment)
Generally, rather than the time interval from the N signal to the S signal (A + B signal) in the first drive mode (mode in which the phase difference cannot be detected), the S signal from the N signal in the second drive mode (mode in which the phase difference can be detected). The time interval until (A + B signal) is long. Therefore, the operating frequency decreases, for example, 1 / f noise increases, and the fluctuation of the power supply that occurs between the AD conversion of the N signal and the AD conversion of the S signal causes a uniform pixel level change. May occur. As a result, there may be a difference in image quality between the pixel signal read in the first drive mode and the pixel signal read in the second drive mode. In this case, if the first drive mode and the second drive mode are mixed in the acquisition of the pixel signal of one frame, a difference in the image for each line occurs in the screen due to the difference in the drive mode, and the entire one frame. The image quality of is deteriorated. In such a case, for example, there is a method of correcting the pixel information read by the second drive mode by using the pixel information read by the first drive mode.

As an example of this correction, consider a 3 × 3 filter as shown in FIG. 12 (a). Such correction is performed by, for example, DSP107. FIG. 12A is an extraction of a part of the pixel array 200. The pixel 1200, which is the unit pixel 201 with dots in the figure, is a correction target pixel, and the surrounding 3 × 3 pixels of the same color are used as correction use pixels 1201 for filtering with respect to the correction target pixel 1200. At this time, assuming that the correction target pixel 1200 is read out in the second drive mode, it is expected that the corrected image quality will be close to the pixel read out by the first drive mode. Therefore, it is desirable that the correction used pixel 1201 uses the pixel information read by the first drive mode.

FIG. 12B shows an example in which the pixel read by the first drive mode cannot be used for the correction of the correction target pixel 1200. Consider the case of correcting the pixels in the n + 2nd row. Considering the 3 × 3 filter processing, the correction used pixels 1101 are the nth row and the nth row + 4th row. However, in the example of FIG. 12B, the nth row and the nth + 4th row are also set to be read in the second drive mode. In this case, even if the pixel information is corrected, the corrected image quality cannot be brought close to that of the first drive mode.

FIG. 12C shows an example in which the pixel read by the first drive mode can be used for the correction of the correction target pixel 1200. Consider the case of correcting the pixels in the n + 3rd row. Considering the 3 × 3 filter processing, the correction used pixels 1201 are the n + 1th row and the n + 5th row. In the example of FIG. 12 (c), both the nth row and the nth + 5th row are set to be read in the first drive mode. In this case, by correcting the pixel information, the corrected image quality can be brought closer to the first drive mode.

Since the purpose of the pixel correction is to bring the image quality of the pixels read in the second drive mode closer to the image quality of the pixels read in the first drive mode, the reading as shown in FIG. 12 (b). One cannot fulfill the purpose of pixel correction. Therefore, when performing pixel correction, the vertical scanning circuit control unit 204 is set so that at least a part of the pixels used for pixel correction is read out by the first drive mode as shown in FIG. 12 (c).

As described above, according to the fourth embodiment, at least a part of the pixels used for the correction of a certain pixel read out by the second drive mode is operated so as to be read out by the first drive mode. Is possible. This makes it possible to correct the pixels read by the second drive mode and bring the image quality closer to the image quality of the pixels read by the first drive mode.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and modifications can be made within the scope of the gist thereof.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100: 1st lens, 101: Aperture, 102: 2nd lens, 103: 3rd lens, 104: Focal plane shutter, 106: Image sensor, 108: RAM, 109: CPU, 110: Display unit, 111: Operation unit , 112: Recording medium, 113: ROM

Claims (28)

An image sensor having a pixel region in which a plurality of unit pixels having one microlens and a plurality of photoelectric conversion units arranged corresponding to the one microlens are arranged in a matrix.
For each pixel in the first region and the second region of the pixel region, the signals of the plurality of photoelectric conversion units are grouped into a first number and the signals are read out in the first mode. It is set whether the plurality of photoelectric conversion units are grouped into a second number larger than the first number and used as a second pixel for reading a signal in a second mode for reading a signal. With the setting means to
The pixels of the first region and the second region are set in the first pixel or the second pixel in row units of the matrix array, and are set in the second pixel. An image pickup apparatus characterized in that the period differs between the first region and the second region . The image pickup apparatus according to claim 1, wherein the first number is 1, and the second number is 2. The image pickup apparatus according to claim 1 or 2 , wherein the first region corresponds to the entire pixel region, and the second region corresponds to a part of the pixel region. .. The image pickup apparatus according to claim 3 , wherein the second region is a focus detection region. The image pickup apparatus according to claim 3 or 4 , wherein the second pixel outputs a signal for focus detection. The image pickup apparatus according to claim 5 , wherein a defocus map is created based on the signal in the first region , and autofocus is performed based on the signal in the second region . The ratio of the number of the second pixels to the number of the first pixels in the first region is the ratio of the number of the second pixels to the number of the first pixels in the second region . The image pickup apparatus according to any one of claims 3 to 6 , wherein the image pickup apparatus is smaller than the above. The claim is characterized in that the row cycle set for the second pixel in the first region is larger than the row cycle set for the second pixel in the second region . 7. The imaging device according to 7. The image pickup apparatus according to any one of claims 1 to 8 , further comprising a changing means for changing the order of signals output from the image pickup element to be different from the order of arrangement of the matrix-shaped pixels. The image pickup apparatus according to claim 9 , wherein the changing means outputs the signal of the first pixel before the signal of the second pixel from the image pickup device. The image pickup apparatus according to claim 9 , wherein the changing means outputs the signal of the second pixel before the signal of the first pixel from the image pickup device. In the region where the first region and the second region overlap each other, when the same row is overlapped in each region and set in the second pixel, the overlapping second region is set. The image pickup apparatus according to any one of claims 1 to 11 , wherein the signal of the pixel set in the pixel is read out only once. The twelfth claim is characterized in that the signal read only once is commonly used for different operations in the first region and the second region using the signal of the second pixel. The imaging device described. The image pickup apparatus according to any one of claims 1 to 13 , further comprising a correction means for correcting the signal of the second pixel by using the signal of the first pixel. A method for controlling an image pickup device including an image pickup device having a pixel region in which a plurality of unit pixels having a plurality of unit pixels having one microlens and a plurality of photoelectric conversion units arranged corresponding to the one microlens are arranged in a matrix. And,
For each pixel in the first region and the second region of the pixel region, the signals of the plurality of photoelectric conversion units are grouped into a first number and the signals are read out in the first mode. It is set whether the plurality of photoelectric conversion units are grouped into a second number larger than the first number and used as a second pixel for reading a signal in a second mode for reading a signal. Has a setting process to
The pixels of the first region and the second region are set in the first pixel or the second pixel in row units of the matrix array, and are set in the second pixel. A control method for an image pickup apparatus, characterized in that the period differs between the first region and the second region . A program for causing a computer to execute each step of the control method according to claim 15 . A computer-readable storage medium that stores a program for causing a computer to execute each step of the control method according to claim 15 . An image sensor having a pixel region in which a plurality of unit pixels having one microlens and a plurality of photoelectric conversion units arranged corresponding to the one microlens are arranged in a matrix.
For each pixel in the first region and the second region of the pixel region, the signals of the plurality of photoelectric conversion units are grouped into a first number and the signals are read out in the first mode. It is set whether the plurality of photoelectric conversion units are grouped into a second number larger than the first number and used as a second pixel for reading a signal in a second mode for reading a signal. With the setting means to
An image pickup apparatus characterized in that a defocus map is created based on a signal in the first region and autofocus is performed based on the signal in the second region. The image pickup apparatus according to claim 18, wherein the first number is 1, and the second number is 2. The imaging device according to claim 18 or 19, wherein the first region corresponds to the entire pixel region, and the second region corresponds to a part of the pixel region. The ratio of the number of the second pixels to the number of the first pixels in the first region is larger than the ratio of the number of the second pixels to the number of the first pixels in the second region. The imaging apparatus according to any one of claims 18 to 20, wherein the image pickup apparatus is small. 21 is characterized in that the row cycle set for the second pixel in the first region is larger than the row cycle set for the second pixel in the second region. The imaging device described. The image pickup apparatus according to any one of claims 18 to 22, further comprising a changing means for changing the order of signals output from the image pickup element to be different from the order of arrangement of the matrix-shaped pixels. 23. The image pickup apparatus according to claim 23, wherein the changing means outputs the signal of the first pixel before the signal of the second pixel from the image pickup device. 23. The image pickup apparatus according to claim 23, wherein the changing means outputs the signal of the second pixel before the signal of the first pixel from the image pickup device. In the region where the first region and the second region overlap each other, when the same row is set to overlap in the second pixel in each region, the overlap is made in the second pixel. The image pickup apparatus according to any one of claims 18 to 25, wherein the signal of the set pixel is read out only once. 26. Claim 26, wherein the signal read only once is commonly used for different operations in the first region and the second region using the signal of the second pixel. Imaging device. The image pickup apparatus according to any one of claims 18 to 27, further comprising a correction means for correcting the signal of the second pixel by using the signal of the first pixel.

Images