JP2018004689A - Image processing device and method, and imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform image processing, without using a large number of parameters, when switching a pupil division direction, without increasing the read-out signal amount of a captured image.SOLUTION: An image processing device processes a focus detection signal and an image signal output from an imaging element comprising: a pixel part including a plurality of photoelectric conversion parts for each of a plurality of microlenses; and read-out control means for reading the focus detection signal used for focus detection and the image signal to which the signals of the plurality of photoelectric conversion parts are added for each of the microlenses from the pixel part by addition read-out or thinning-out read-out according to a control signal. The image processing device includes conversion means for converting the image signal read from the imaging element into the image signal of an image having a predetermined size on the basis of the control signal.SELECTED DRAWING: Figure 6

Description

  The present invention relates to an image processing device and method, and an imaging device, and more particularly to a technique for processing a signal obtained from an imaging device having a plurality of photoelectric conversion units for one microlens.

  Conventionally, a pupil divided image and a picked-up image are obtained using an image pickup element configured by unit pixel cells having a plurality of photoelectric conversion units for one microlens, and the phase difference of the obtained pupil divided image is calculated. Techniques for performing focus detection in search are known.

  Patent Document 1 discloses a technology that enables generation of a pupil-division image and a captured image in a predetermined direction in an image sensor by sharing a floating diffusion unit (FD) among a plurality of photoelectric conversion units in a unit pixel cell. Has been. By using such a configuration, it is possible to efficiently transfer the captured image and the pupil-divided image while saving the transmission path.

  Further, if the technique described in Patent Document 1 is applied and the FD is similarly shared between the unit pixel cells, signal addition in a wider range can be performed. By doing so, it is possible to reduce the signal transfer amount when shooting a high frame rate moving image or when the number of photoelectric conversion units for one microlens is increased.

JP 2014-157338 A

  However, when a signal sharing is performed within the image sensor and the captured image and the pupil-divided image in a predetermined direction are transferred using the configuration sharing the FD, the image size is set in the division direction of the pupil-divided image. It will change accordingly. For this reason, there arises a problem that the output image signal is not suitable for a component such as an output panel that requires input of a certain image size.

  In addition, when correcting deterioration of a captured image due to deterioration of characteristics of the image sensor such as shading, there arises a problem that different correction parameters must be prepared depending on the division direction.

  The present invention has been made in view of the above problems, and enables image processing without using a large number of parameters when switching the pupil division direction without increasing the readout signal amount of a captured image. Objective.

  To achieve the above object, a pixel unit including a plurality of photoelectric conversion units for each of a plurality of microlenses, a focus detection signal used for focus detection from the pixel unit according to a control signal, and The present invention processes a focus detection signal and an image signal output from an image pickup device including a read control unit that reads an image signal obtained by adding the signals of the plurality of photoelectric conversion units for each microlens by addition reading or thinning readout. The image processing apparatus includes conversion means for converting an image signal read from the image sensor into an image signal of an image having a predetermined size based on the control signal.

  According to the present invention, when the pupil division direction is switched without increasing the readout signal amount of the captured image, it is possible to perform image processing without using a large number of parameters.

1 is a block diagram showing a schematic configuration of an imaging apparatus according to a first embodiment of the present invention. FIG. 3 is a top view of a pixel portion of the image sensor according to the first embodiment. FIG. 3 is a circuit diagram illustrating a configuration example of a pixel unit in the first embodiment. FIG. 3 is a conceptual diagram of signal addition and output in the image sensor according to the first embodiment. The circuit diagram inside the addition signal separation part in a 1st embodiment. The block diagram which shows the internal structure of the image adjustment part in 1st Embodiment. The figure which showed the intermediate signal by the image adjustment part in 1st Embodiment. The figure explaining the relationship between a pupil division image and a focus state.

  DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings.

<First Embodiment>
FIG. 1 is a block diagram showing a configuration of an imaging apparatus according to the first embodiment of the present invention. In FIG. 1, an imaging optical system 101 includes a focus lens, a zoom lens, a diaphragm, and the like for adjusting the focus position. The optical system drive unit 102 performs drive control of the imaging optical system 101 according to the defocus amount output from the phase difference focus detection unit 108 and the imaging optical system drive information output from the system control unit 109.

  The image sensor 103 photoelectrically converts a light beam incident via the image pickup optical system 101 based on a drive control command from the image sensor drive unit 104, and displays a pupil-divided image (focus detection signal) used for focus detection, display and recording. A captured image to be used is read out. Based on the pupil division direction information output from the system control unit 109, the image sensor driving unit 104 outputs a drive control command such as signal reading of the image sensor 103.

  The addition signal separation unit 105 subtracts the pupil-divided image from the captured image output from the image sensor 103 to generate a signal of the other pupil-divided image, and the captured image and the signal of the pair of pupil-divided images are scratched at the subsequent stage. It outputs to the correction | amendment part 110 and the phase difference focus detection part 108, respectively.

  The defect correction unit 110 selectively reads out correction parameters based on the pupil division direction information output from the system control unit 109, and includes the captured image and the pair of pupil division image signals output from the addition signal separation unit 105. Correct the included scratch data. That is, scratch correction is performed using different correction parameters depending on the pupil division direction.

  The image adjustment unit 106 adjusts the captured image output from the scratch correction unit 110 to a predetermined image size based on the pupil division direction information output from the system control unit 109 and outputs the image.

  The shading correction unit 107 multiplies a predetermined parameter determined in advance with the signal of the captured image, and corrects a peripheral light amount drop in the captured image. Here, the shading correction unit 107 performs shading correction using a common correction parameter regardless of the pupil division direction.

  Based on the pupil division direction information output from the system control unit 109, the phase difference focus detection unit 108 performs the phase difference calculation of the signals of the pair of pupil division images output from the addition signal separation unit 105, and calculates the defocus amount. calculate.

  The system control unit 109 controls the entire imaging apparatus, determines a pupil division direction based on shooting information obtained from a user instruction, shooting scene detection, subject detection, and the like, and includes an image sensor driving unit 104, an image adjustment unit 106, The pupil division direction information is output to the phase difference focus detection unit 108.

  Next, details of the image sensor 103 will be described. FIG. 2 is a top view schematically showing the pixel portion of the image sensor 103 according to the first embodiment. In FIG. 2, the unit pixel cell 200 includes photoelectric conversion units PDnA to PDnA to D represented by photodiodes for each microlens 201 (n = 1 to 4 in FIG. 2). Note that the photoelectric conversion units PDnA to PDnD exist at the same position as the unit pixel cell 200 for other unit pixel cells existing on the image sensor 103.

  Here, a signal obtained by signal addition in the unit pixel cell 200 will be described. First, by adding signals obtained from the photoelectric conversion units PDnA and PDnB, a pupil division image (upper signal) of the upper region when the exit pupil is divided in the vertical direction is obtained. Further, by adding the signals obtained from the photoelectric conversion units PDnC and PDnD, a pupil division image (lower signal) of the lower region when the exit pupil is divided in the vertical direction is obtained.

  Further, by adding the signals obtained from the photoelectric conversion units PDnA and PDnC, a pupil division image (left signal) in the left region when the exit pupil is divided in the horizontal direction is obtained. Then, by adding the signals obtained from the photoelectric conversion units PDnB and PDnD, a pupil division image (right signal) in the right region when the exit pupil is divided in the horizontal direction is obtained.

  And a picked-up image can be obtained by adding all the values obtained from photoelectric conversion part PDnA-D.

  In this way, by changing the combination of addition of the photoelectric conversion units in the unit pixel cell 200, it is possible to obtain upper, lower, left, and right pupil divided images and captured images.

  FIG. 3 is a diagram illustrating a configuration example of the image sensor 103 according to the first embodiment of the present invention. FIG. 3 shows photoelectric conversion units (photodiodes) PD1A to PD1 and PD2A to D corresponding to n = 1 and 2 among the photoelectric conversion units PDnA to D shown in FIG. Hereinafter, “PD” is simply added unless it is necessary to specify which photoelectric conversion unit. In each photoelectric conversion unit PD, charges are transferred to the floating diffusion unit FD by the read transistors RdnA to Rdn.

  The power supply VDD is a power supply for the image sensor 103 and clears the electric charge of the floating diffusion portion FD via the reset transistor Res. The row readout transistor Sel is a switch that selects a row from which charges are read, and a pixel value corresponding to the charge in the selected row is output via the source follower.

  Next, a control method in the case of obtaining a pupil divided image and a captured image by signal addition in the unit pixel cell 200 will be described.

  For example, when the upper signal is obtained, the pixel values are reset by the reset transistor Res, and then the read transistors RdnA and RdnB are controlled, and the charges of the photoelectric conversion units PDnA and PDnB are transferred to the floating diffusion unit FD. Next, the upper signal can be obtained by controlling the row readout transistor Sel and outputting the pixel value to an A / D converter (not shown).

  Further, when obtaining a pupil division image in another direction, the combination of readout transistors is appropriately changed in the same procedure, and the charge of the photoelectric conversion unit PD constituting the pupil division image in a desired direction is transferred to the floating diffusion unit FD. That's fine.

  In addition, when obtaining a captured image, all of the readout transistors RdnA to DD are controlled in the same procedure, and the charges of the photoelectric conversion units PDnA to Dn may be transferred to the floating diffusion unit FD.

  Thus, by controlling the combination of the read transistors RdnA to D in the unit pixel cell 200, it is possible to freely add signals in the unit pixel cell 200. Thereby, a pupil division image and a picked-up image in a predetermined direction can be generated in the image pickup device 103.

  Note that the image sensor 103 according to the first embodiment uses a configuration in which the adjacent unit pixel cell 200 shares the floating diffusion portion FD. Thereby, in addition to the signal addition in the unit pixel cell 200 described above, the signal addition between the unit pixel cells 200 can be performed.

  Next, a specific control method for obtaining a pupil-divided image and a captured image by signal addition in the unit pixel cell 200 including the photoelectric conversion units PD1A to D and the unit pixel cell 200 including the photoelectric conversion units PD2A to D will be specifically described. explain.

  For example, when the signal addition of the upper signal is performed between the two unit pixel cells 200 that are aired, the pixel value is first reset by the reset transistor Res. Thereafter, the readout transistors Rd1A and Rd1B and the readout transistors Rd2A and Rd2B are controlled, and the charges of the photoelectric conversion units PD1A, PD1B, PD2A, and PD2B are transferred onto the floating diffusion portion FD. Next, the row readout transistor Sel is controlled to output the pixel value to an A / D conversion unit (not shown).

  In addition, when obtaining the pupil division image and captured image of another direction, what is necessary is just to change suitably the combination of reading transistor RdnA-D in the same procedure. In each unit pixel cell sharing the floating diffusion portion FD as described above, the signal addition between the unit pixel cells 200 can be performed by controlling each readout transistor RdnA to D.

  Here, the description has been made with reference to FIG. 3 using a configuration in which two unit pixel cells 200 adjacent in the horizontal direction share an FD. However, since the image sensor 103 of the first embodiment also shares the floating diffusion portion FD for the unit pixel cells 200 adjacent in the vertical direction, signal addition between the unit pixel cells 200 in the vertical direction is also performed. Is possible. The various controls described here are realized by the image sensor driving unit 104.

  The image sensor driving unit 104 receives 1-bit pupil division direction information indicating the pupil division direction from the system control unit 109. When the pupil division direction information is 0, the image sensor driving unit 104 performs signal addition between the unit pixel cells 200 adjacent to each other in the vertical direction on one pupil division image and the picked-up image of the region divided in the horizontal direction. The driving of the image sensor 103 is controlled so as to output the above. When the pupil division direction information is 1, the image sensor drive unit 104 adds a signal between the unit pixel cells 200 adjacent in the horizontal direction to one pupil divided image and the picked-up image of the region divided in the vertical direction. The drive of the image sensor 103 is controlled so that the output is performed after the execution.

  By controlling to add signals between the unit pixel cells 200 in the direction orthogonal to the pupil division direction in this way, it is possible to reduce the transfer time from the image sensor 103 while maintaining the resolution of the pupil division image. It becomes.

  FIG. 4 is a conceptual diagram of signal addition and output in the image sensor 103 according to the first embodiment. Here, a description will be given by taking, as an example, six horizontal pixels and four vertical pixels of the image sensor 103. Note that the signal addition between the unit pixel cells 200 is performed so that the captured image can be suppressed to a half of the total pixel readout.

  FIG. 4A illustrates an example of signal addition of a left signal (a signal from the photoelectric conversion unit PDnA + PDnC) as one of the regions obtained by pupil division in the horizontal direction to obtain a pupil division image when the pupil division direction information is 0. FIG. When performing pupil division in the horizontal direction, the readout transistors RdnA and RdnC are controlled so that the combination of signal addition in the unit pixel cell 200 is a left signal. Similarly, the read transistors RdnA and RdnC are controlled for the unit pixel cells 200 adjacent in the vertical direction. By performing the control in this way, the charge of the left signal of two unit pixel cells 200 adjacent in the vertical direction is transferred to the floating diffusion portion FD, and the left signal subjected to signal addition can be acquired.

  FIG. 4B is a diagram illustrating an example in which the left signal is output when the pupil division direction information is zero. As the output signal, since signal addition is performed between the vertical unit pixel cells 200, finally, a left signal constituting 6 horizontal pixels and 2 vertical pixels of the pupil divided image is output.

  On the other hand, FIG. 4C is a diagram illustrating a signal addition example of a captured image (photoelectric conversion unit PDnA + PDnB + PDnC + PDnD signal) when pupil division direction information is zero. Here, as shown in FIG. 4A, after the left signal is read out, the floating diffusion unit FD is not reset, and the remaining right signals (signals of the photoelectric conversion units PDnB + PDnD) are read out transistors RdnB and RdnD. It is generated by controlling. That is, a captured image is generated by transferring the right signal charge to the floating diffusion portion FD while the left signal charge remains.

  FIG. 4D is a diagram illustrating an example in which a captured image is output when the pupil division direction information is zero. As the output signal, signal addition between the unit pixel cells 200 is performed in the vertical direction in the same manner as the left signal, so that finally, signals constituting the horizontal 6 pixels and the vertical 2 pixels of the captured image are output.

  In this way, by reading out one pupil-divided image and then performing control so as to read the captured image, the signal reading time can be reduced.

  FIG. 4E shows an example of signal addition of an upper signal (a signal from the photoelectric conversion unit PDnA + PDnB) as one of the regions obtained by pupil division in the vertical direction to obtain a pupil division image when the pupil division direction information is 1. FIG. When performing pupil division in the vertical direction, the readout transistors RdnA and RdnB are controlled so that the combination of signal addition in the unit pixel cell 200 becomes an upper signal. Similarly, the read transistors RdnA and RdnB are controlled for the unit pixel cells 200 adjacent in the horizontal direction. By performing the control in this way, the charge of the upper signal of the two unit pixel cells 200 adjacent in the horizontal direction is transferred to the floating diffusion portion FD, and the upper signal subjected to signal addition can be acquired.

  FIG. 4F is a diagram illustrating an example in which the upper signal is output when the pupil division direction information is 1. As an output signal, since signal addition is performed between the unit pixel cells 200 in the horizontal direction, finally, an upper signal constituting three horizontal pixels and four vertical pixels of the pupil divided image is output.

  On the other hand, FIG. 4G is a diagram illustrating a signal addition example of a captured image (photoelectric conversion unit PDnA + PDnB + PDnC + PDnD signal) when the pupil division direction information is 1. Here, as shown in FIG. 4E, after reading the upper signal, the floating diffusion unit FD is not reset, and the remaining lower signals (photoelectric conversion units PDnC + PDnD signals) are read by the transistors RdnC and RdnD. It is generated by controlling. That is, a captured image is generated by transferring the lower signal charge to the floating diffusion portion FD while the upper signal charge remains.

  FIG. 4D is a diagram illustrating an example in which a captured image is output when the pupil division direction information is 1. As the output signal, signal addition between the unit pixel cells 200 is performed in the horizontal direction in the same manner as the upper signal, so that finally, signals constituting the horizontal 3 pixels and the vertical 4 pixels of the captured image are output.

  One of the read pupil divided images and the captured image is transferred to the subsequent addition signal separation unit 105, and the other one of the pupil division images is calculated by arithmetic processing inside the addition signal separation unit 105.

  Next, a detailed operation of the addition signal separation unit 105 will be described. FIG. 5 is a circuit diagram inside the addition signal separation unit 105 in the first embodiment. The addition signal separation unit 105 generates a second pupil divided image by subtracting the first pupil divided image from the input captured image, and the captured image, the first pupil divided image, and the second pupil divided image. Are output respectively.

  Note that the input captured image and the first pupil-divided image are input to the addition signal separation unit 105 in a synchronized state via a delay adjustment unit (not shown). As the first pupil division image, either the upper signal (A + B signal) or the left signal (A + C signal) is selectively input according to the pupil division direction.

  FIG. 5A is a diagram for explaining the flow of signal processing performed by the addition signal separation unit 105 with respect to the signal read from the image sensor 103 when the pupil division direction information is zero. When a captured image (photoelectric conversion unit PDnA + PDnB + PDnC + PDnD signal) and a left signal (photoelectric conversion unit PDnA + PDnC signal) are input from the image sensor 103, the left signal is subtracted from the captured image, and the right signal (photoelectric conversion unit PDnB + PDnD signal). ) Is obtained. Then, the added signal separation unit 105 outputs the captured image, the left signal, and the right signal to the subsequent stage.

  FIG. 5B is a diagram for explaining a flow of signal processing performed by the addition signal separation unit 105 for a signal read from the image sensor 103 when the pupil division direction information is 1. In this case, a picked-up image (photoelectric conversion unit PDnA + PDnB + PDnC + PDnD signal) and an upper signal (photoelectric conversion unit PDnA + PDnB signal) are input from the image sensor 103. Therefore, the upper signal is subtracted from the captured image, and a lower signal (photoelectric conversion unit PDnC + PDnD signal) is obtained. The added signal separation unit 105 outputs the captured image, the upper signal, and the lower signal to the subsequent stage.

  In this way, it is possible to obtain the other pupil divided image by subtracting one pupil divided image from the captured image.

  Next, a detailed operation of the image adjustment unit 106 will be described. FIG. 6 is a block diagram illustrating an internal configuration of the image adjustment unit 106 according to the first embodiment. The image adjustment unit 106 includes a horizontal adjustment unit 601, a vertical adjustment unit 602, and a normalization unit 603.

  The horizontal adjustment unit 601 adjusts the size of the input signal 604, which is a captured image, in the horizontal direction based on the pupil division direction information. Specifically, when the pupil division direction information is 0, that is, when the pupil division is performed in the horizontal direction, the horizontal adjustment unit 601 controls to add and output the captured image in units of two pixels in the horizontal direction. On the other hand, when the pupil division direction information is 1, that is, when the pupil division is performed in the vertical direction, the horizontal adjustment unit 601 performs control so that the captured image is output through. An output signal of the horizontal adjustment unit 601 is an intermediate signal 605.

  The vertical adjustment unit 602 adjusts the size of the captured image (the intermediate signal 605) in the vertical direction based on the pupil division direction information. Specifically, when the pupil division direction information is 0, that is, when the pupil division is performed in the horizontal direction, the vertical adjustment unit 602 performs control so that the captured image is output through. On the other hand, when the pupil division direction information is 1, that is, when the pupil division is performed in the vertical direction, the vertical adjustment unit 602 performs control so that the captured images are added and output in units of two pixels in the vertical direction. An output signal of the vertical adjustment unit 602 is an intermediate signal 606.

  The intermediate signal 606 whose size has been adjusted by the horizontal adjustment unit 601 and the vertical adjustment unit 602 is normalized to a predetermined signal level by the normalization unit 603 and output. The normalization unit 603 shifts the input signal to the right by 2 bits regardless of the pupil division direction information, and outputs the signal level by 1/4.

  The above operation will be further described with reference to FIG. The upper part of FIG. 7 shows an example when the pupil division direction information is 0, that is, when the pupil division is performed in the horizontal direction. It is assumed that the input captured image is obtained from the image sensor 103 as described above with reference to FIG. In this case, as shown in FIG. 4 (d), as the input signal 604, signals constituting the horizontal 6 pixels and the vertical 2 pixels of the captured image are input from the area of the horizontal 6 pixels and the vertical 4 pixels. Since the input captured image is subjected to two-pixel addition in the horizontal direction by the horizontal adjustment unit 601, a signal constituting three horizontal pixels and two vertical pixels of the captured image is output as the intermediate signal 605. The intermediate signal 605 is input to the vertical adjustment unit 602 and output as it is. Therefore, as the intermediate signal 606, a signal constituting three horizontal pixels and two vertical pixels of the captured image is output.

  On the other hand, the lower part of FIG. 7 shows an example when the pupil division direction information is 1, that is, when the pupil division is performed in the vertical direction. Assume that the image sensor that generates the input captured image is obtained from the image sensor 103 as described above with reference to FIG. In this case, as shown in FIG. 4 (h), as the input signal 604, signals constituting the horizontal 3 pixels and the vertical 4 pixels of the captured image are input from the region of 6 horizontal pixels and 4 vertical pixels. The input captured image passes through the horizontal adjustment unit 601 as it is, and a signal constituting the horizontal 3 pixels and the vertical 4 pixels of the captured image is output as the intermediate signal 605. The intermediate signal 605 is input to the vertical adjustment unit 602, and two pixels are added in the vertical direction. Therefore, as the intermediate signal 606, a signal constituting three horizontal pixels and two vertical pixels of the captured image is output.

  Accordingly, as the intermediate signal 606, a captured image having a size of three horizontal pixels and two vertical pixels is output in any pupil division direction.

  By switching the image adjustment control in accordance with the pupil division direction in this way, it becomes possible to output a captured image reduced at a constant reduction rate, and in the processing after the image adjustment unit 106, for each captured image size. It is no longer necessary to prepare a large number of parameters.

  Here, the image adjustment unit 106 has been described with the horizontal adjustment unit 601 and the vertical adjustment unit 602 configured, but the present invention is not limited to this. The internal configuration and operation may be changed as appropriate as long as the configuration allows the output captured image to be adjusted and controlled so as to have a certain size regardless of the reading method of the image sensor 103.

  FIG. 8 is a diagram for explaining the relationship between the pupil divided image and the focus state. In FIG. 8A, reference numeral 800 shows the exit pupil of the imaging optical system 101. Here, a case where the pupil division direction is the horizontal direction will be described. The light beam incident on the imaging optical system 101 passes through the first region 801 and the second region 802 of the exit pupil 800, and the photoelectric conversion unit PDnA and the photoelectric conversion unit PDnC of the pixels configured in the imaging element 103, respectively. It is led to the conversion unit PDnB and the photoelectric conversion unit PDnD.

  An image formed by a signal (left signal) group acquired by the photoelectric conversion unit PDnA and the photoelectric conversion unit PDnC, and an image formed by a signal (right signal) group acquired by the photoelectric conversion unit PDnB and the photoelectric conversion unit PDnD. Can be handled as a pair of pupil-divided images that are divided into left and right pupils.

  FIG. 8A shows the state of the pupil divided image by the left signal group and the right signal group in the focused state. In the in-focus state, the shapes of the divided pupil images of the left signal group and the right signal group are substantially the same.

  FIG. 8B illustrates a state of a pupil divided image by the left signal group and the right signal group in the front pin state. In the front pin state, the pupil division images of the left signal group and the right signal group are in a relatively shifted state.

  FIG. 8C illustrates a state of a pupil divided image by the left signal group and the right signal group in the rear pin state. In the rear pin state, the pupil-divided images of the left signal group and the right signal group are in a relatively shifted state, and the direction of shift at this time is the same as that in the front pin state shown in FIG. Is in the opposite direction.

  Next, details of the phase difference focus detection unit 108 will be described. The phase difference focus detection unit 108 receives, from the addition signal separation unit 105, either a pair of pupil division images composed of a left signal and a right signal, or a pair of pupil division images composed of an upper signal and a lower signal, and performs correlation calculation processing. I do. As the correlation calculation processing, for example, a SAD (Sum of Abusolute Difference) method can be used. In the SAD method, an absolute value of a difference for each pixel of a pair of pupil divided images is calculated, and a correlation evaluation value between the pair of pupil divided images is obtained by obtaining the sum thereof.

  The correlation evaluation values calculated in this way are executed in parallel (or repeatedly) while changing the relative positions of the pair of pupil-divided images, thereby acquiring the correlation evaluation values at the respective relative positions. Then, the amount of defocus is determined by detecting the relative image shift amount up to the position with the highest correlation and multiplying the detected relative image shift amount up to the position with the highest correlation by a predetermined coefficient. Is calculated. The calculated defocus amount is output to the optical system driving unit 102.

  Although an example using the SAD method has been described here for explanation, a known correlation operation such as SSD (Sum of Squared Difference) or NCC (Normalized Cross-Correlation) may be used.

  As described above, according to the first embodiment, the image adjustment unit that adjusts the size of the captured image read by the image sensor is configured in the subsequent stage of the image sensor. As a result, when the pupil division direction is switched without increasing the readout signal amount of the captured image, image processing for correcting the deterioration of the captured image due to the deterioration of the characteristics of the image sensor is performed without using a large number of parameters. It becomes possible.

  In the first embodiment, an example in which the read signal amount is reduced by adding signals between unit pixel cells is used. However, the same applies to the case where the signal amount is reduced by thinning processing instead of signal addition between pixels. An effect can be obtained.

  In the first embodiment, an example in which the scratch correction unit 110 is configured has been described. However, the present invention is not limited to the scratch correction unit 110, and may be configured to correct scratches and noise caused by the image sensor itself. Thus, the effects of the present invention can be obtained.

  In the first embodiment, the example in which the shading correction unit 107 is configured has been described. However, the present invention is not limited to the shading correction unit 107. The effect of the present invention can be obtained as long as correction is performed by a correction method whose characteristics change depending on the image height or the like.

<Second Embodiment>
Hereinafter, a second embodiment of the present invention will be described. In the first embodiment described above, the example in which the reading of the image sensor 103 and the control of the image adjustment unit are switched in accordance with the pupil division direction has been described. In contrast, in the second embodiment, an example in which the readout method and the image adjustment unit are controlled based on the correlation calculation method in the phase difference focus detection unit 108 will be described. The configuration of the imaging apparatus is the same as that described in the first embodiment with reference to FIGS.

  In the first embodiment, it has been described that the phase difference can be detected by performing the calculation while changing the relative position of the pupil divided image by the phase difference focus detection unit 108. At this time, it is ideal if the correlation evaluation values at all the relative positions can be obtained with the highest resolution using the signals that can be acquired by the image sensor 103, but in order to obtain the evaluation values at all the relative positions. Increases the amount of computation, and causes problems such as an increase in circuit scale and computation processing time.

  In order to solve this problem, a method of performing phase difference focus detection calculation step by step by switching between coarse and fine calculations can be considered. Coarse phase difference focus detection calculation reduces the number of signals by increasing the number of additions in addition readout (in the case of thinning readout, increasing the thinning rate) and lowering the resolution of the pupil-divided image, and correlation in the low frequency band To ask. On the other hand, the dense phase difference ranging calculation does not change the number of addition readings (thinning rate in the case of thinning readout), so that the combination of relative positions (search range) can be obtained without reducing the resolution of the pupil-divided image. By limiting, the correlation of the high frequency band is obtained. If the calculation method is controlled step by step so that rough focus adjustment is performed with coarse phase difference focus detection calculation, and then more precise focus adjustment is performed with fine phase difference focus detection calculation, phase difference focus detection is performed. High-precision focus detection can be realized without increasing the calculation load in the unit 108.

  Further, from the viewpoint of reading from the image sensor 103, it is preferable to perform reading after adding signals in the image sensor 103 in different manners depending on the calculation method of phase difference focus detection. However, similarly to the first embodiment described above, there arises a problem that the size of the image signal read from the image sensor 103 changes.

  Therefore, in the imaging apparatus according to the second embodiment, instead of the pupil division direction information used in the first embodiment, either coarse phase difference focus detection calculation or dense phase difference focus detection calculation is used. 1-bit calculation method information defining the above is output. Since the control signal for determining the drive of the image sensor 103 and the control of the image adjustment unit 106 is merely changed from the pupil division direction information to the calculation method information, the image signal read out in accordance with each calculation method has a certain size. The image adjustment unit 106 may be controlled so as to make adjustments.

<Other embodiments>
Note that the present invention may be applied to a system composed of a plurality of devices or an apparatus composed of a single device.

  Further, the present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus execute the program. It can also be realized by a process of reading and executing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  DESCRIPTION OF SYMBOLS 101: Imaging optical system, 102: Optical system drive part, 103: Image sensor, 104: Image sensor drive part, 105: Addition signal separation part, 106: Image adjustment part, 107: Shading correction part, 108: Phase difference focus detection 109: System control unit 110: Scratch correction unit 200: Unit pixel cell 201: Micro lens PDnA to D: Photoelectric conversion unit RdnA to D: Read transistor FD: Floating diffusion unit

Claims (22)

A pixel unit including a plurality of photoelectric conversion units for each of the plurality of microlenses, a focus detection signal used for focus detection from the pixel unit, and the plurality of photoelectric elements for each microlens according to a control signal. An image processing apparatus for processing a focus detection signal and an image signal output from an image pickup device, comprising: a readout control unit that reads out an image signal obtained by adding signals from a conversion unit by addition readout or thinning readout;
An image processing apparatus comprising: conversion means for converting an image signal read from the image pickup device into an image signal of an image having a predetermined size based on the control signal. Comprising at least one of a first correction unit and a second correction unit for correcting degradation of a captured image caused by the characteristics of the image sensor;
The first correction unit corrects an image before the size is converted by the conversion unit, and the second correction unit corrects an image after the size is converted by the conversion unit. The image processing apparatus according to claim 1, wherein: The first correction unit performs correction using a different correction parameter for each of the different control signals,
The image processing apparatus according to claim 2, wherein the second correction unit performs correction using a common correction parameter regardless of the control signal.   The image processing apparatus according to claim 1, further comprising a focus detection unit configured to detect a focus state based on the focus detection signal.   The image processing apparatus according to claim 1, wherein the control signal indicates a direction in which the plurality of photoelectric conversion units are divided.   When the control signal indicates division in the first direction, the readout control unit is configured to correspond to the plurality of photoelectric conversion units respectively corresponding to the plurality of microlenses adjacent in the second direction orthogonal to the first direction. 6. The image processing apparatus according to claim 5, wherein the image processing apparatus is divided into two in the first direction, and the signals of at least one of the divided photoelectric conversion units are added and read out as the focus detection signal. When the control signal indicates division in the first direction, the readout control unit is configured to correspond to the plurality of photoelectric conversion units respectively corresponding to the plurality of microlenses adjacent in the second direction orthogonal to the first direction. Are read out as the image signal,
The image processing apparatus according to claim 6, further comprising a calculation unit that obtains the other focus detection signal by subtracting the one focus detection signal from the image signal. The converting means includes first adjusting means for adjusting the image size in the first direction, and second adjusting means for adjusting the image size in the second direction,
When the control signal indicates division in the first direction, adjustment is performed by the first adjustment unit, and when the control signal indicates division in the second direction, adjustment is performed by the second adjustment unit. The image processing apparatus according to claim 7, wherein the image processing apparatus is performed. The focus detection means can detect a focus state using a plurality of correlation calculation methods,
The image processing apparatus according to claim 4, wherein the control signal indicates one of the plurality of correlation calculation methods.   The image processing apparatus according to claim 9, wherein the read control unit changes an addition number of the addition reading or a thinning rate of the thinning reading according to the correlation calculation method. A pixel unit including a plurality of photoelectric conversion units for each of a plurality of microlenses, a focus detection signal used for focus detection from the image sensor according to a control signal, and the plurality of photoelectric conversions for each microlens A reading control means for reading out an image signal obtained by adding the signals of the units by addition or thinning-out reading, and an image processing device according to any one of claims 1 to 10, An imaging device. A pixel unit including a plurality of photoelectric conversion units for each of the plurality of microlenses, a focus detection signal used for focus detection from the pixel unit, and the plurality of photoelectric elements for each microlens according to a control signal. An image processing method for processing a focus detection signal and an image signal output from an image pickup device, comprising: a read control unit that reads an image signal obtained by adding signals from a conversion unit by addition reading or thinning readout;
An image processing method comprising: a conversion step in which a conversion unit converts an image signal read from the image sensor into an image signal of an image having a predetermined size based on the control signal.   13. The image according to claim 12, further comprising a correction step of correcting the deterioration of the captured image caused by the characteristics of the image sensor on the image whose size has been converted by the conversion unit. Processing method.   The image processing method according to claim 13, wherein the correction unit performs correction using a common correction parameter regardless of the control signal.   15. The image processing method according to claim 12, further comprising a focus detection step in which the focus detection unit detects a focus state based on the focus detection signal.   The image processing method according to claim 12, wherein the control signal indicates a direction in which the plurality of photoelectric conversion units are divided. When the control signal indicates division in the first direction, the readout control unit is configured to correspond to the plurality of photoelectric conversion units respectively corresponding to the plurality of microlenses adjacent in the second direction orthogonal to the first direction. Is divided into two in the first direction, the signals of at least one of the divided photoelectric conversion units are added and read out as the focus detection signal, and in the second direction orthogonal to the first direction. Add the signals of the plurality of photoelectric conversion units corresponding respectively to a plurality of adjacent microlenses, and read out as the image signal,
17. The image processing method according to claim 16, further comprising a calculation step of calculating the other focus detection signal by subtracting the one focus detection signal from the image signal. The conversion step includes
A first adjustment step of adjusting an image size in the first direction when the control signal indicates division in a first direction;
The image processing method according to claim 17, further comprising: a second adjustment step of adjusting an image size in the second direction when the control signal indicates division in the second direction. In the focus detection step, it is possible to detect a focus state using a plurality of correlation calculation methods,
The image processing method according to claim 15, wherein the control signal indicates one of the plurality of correlation calculation methods.   The image processing apparatus according to claim 19, wherein the read control unit changes an addition number of the addition reading or a thinning rate of the thinning reading according to the correlation calculation method.   A program for causing a computer to execute each step of the image processing method according to any one of claims 12 to 20.   A computer-readable storage medium storing the program according to claim 21.

Images