JP6362070B2 - Image processing apparatus, imaging apparatus, image processing method, program, and storage medium - Google Patents

Description

  The present invention relates to an imaging apparatus including a plurality of photoelectric conversion units sharing one microlens.

  2. Description of the Related Art Conventionally, an imaging apparatus including an imaging element including a plurality of photoelectric conversion units (hereinafter referred to as “PD”) for one micro lens (hereinafter referred to as “ML”) is known. Such an imaging apparatus generates a plurality of pupil division images based on signals from a plurality of photoelectric conversion units, and performs focus detection (phase difference type focus detection) based on a phase difference between the plurality of pupil division images. Do. However, in the phase difference type focus detection, correct focus detection may be difficult due to the influence of shading.

  Patent Document 1 discloses a method for performing appropriate shading correction for each pupil division image. However, in Patent Document 1, since saturated pixels are not taken into consideration, the focus detection accuracy may be lowered by performing shading correction. On the other hand, Patent Document 2 discloses a configuration in which shading correction is not performed on saturated pixels when shading correction is performed with a different correction amount for each pupil divided image.

JP 2011-223562 A JP-A 63-276010

  By the way, in the configuration provided with a plurality of PDs (divided PDs) sharing one ML, the charge storage capacity per PD is reduced. When at least one divided PD is saturated, even if outputs of a plurality of PDs sharing the same ML are added, a linear sensitivity characteristic cannot be obtained due to the influence of saturation, and the image quality deteriorates.

  In addition, when performing phase difference type focus detection, the method of Patent Document 2 does not use saturated pixels, so that it is possible to avoid the influence of unnecessary correction as in Patent Document 1. However, since saturated pixels cannot be used for focus detection, focus detection accuracy is reduced. Even if saturated pixels are used for focus detection, the focus detection accuracy decreases.

  Therefore, the present invention can provide an image processing device, an imaging device, an image processing method, a program, and a storage medium that can improve focus detection accuracy and acquire a high-quality image.

An image processing apparatus according to one aspect of the present invention is an image processing apparatus that processes an image signal output from an imaging element including a first photoelectric conversion unit and a second photoelectric conversion unit sharing one microlens. Determination means for determining whether each of the first signal obtained from the first photoelectric conversion unit and the second signal obtained from the second photoelectric conversion unit has reached a predetermined level; Calculation for calculating the amount of parallax between the correction unit that corrects the first signal or the second signal based on information from the determination unit, and the first photoelectric conversion unit and the second photoelectric conversion unit And gain setting means for reducing the level of the signal including color information obtained from the corrected first signal and second signal based on the information from the determination means and the information from the calculation means have a, said Gay Setting means, when one of the first signal and the second signal has reached the predetermined level, based on the parallax amount, reduces the level of the signal including the color information.

An imaging apparatus according to another aspect of the present invention includes an imaging device including a first photoelectric conversion unit and a second photoelectric conversion unit sharing one microlens, and a first obtained from the first photoelectric conversion unit. And the second signal obtained from the second photoelectric conversion unit determine whether or not each has reached a predetermined level, and the first signal based on information from the determination unit Or a correction unit that corrects the second signal, a calculation unit that calculates a parallax amount between the first photoelectric conversion unit and the second photoelectric conversion unit, and information from the determination unit and the calculation unit on the basis of the information, have a gain setting means for reducing the level of a signal including a color information obtained from the first signal and the second signal after the correction, the gain setting means, said first One of the signal and the second signal If is has reached the predetermined level, based on the parallax amount, reduces the level of the signal including the color information.

An image processing method according to another aspect of the present invention is an image processing method for processing an image signal output from an imaging device including a first photoelectric conversion unit and a second photoelectric conversion unit sharing one microlens. A determination step of determining whether each of the first signal obtained from the first photoelectric conversion unit and the second signal obtained from the second photoelectric conversion unit has reached a predetermined level; , A correction step for correcting the first signal or the second signal based on the information obtained in the determination step, and a parallax amount between the first photoelectric conversion unit and the second photoelectric conversion unit And a color information obtained from the corrected first signal and the second signal based on the information obtained in the determination step and the information obtained in the calculation step. Signal level Possess a setting step of setting the gain to reduce the setting step, when one of the first signal and the second signal has reached the predetermined level, based on the amount of parallax The level of the signal including the color information is reduced .

  A program according to another aspect of the present invention is configured to cause a computer to execute the image processing method.

  A storage medium according to another aspect of the present invention stores the program.

  Other objects and features of the invention are described in the following embodiments.

  According to the present invention, it is possible to provide an image processing device, an imaging device, an image processing method, a program, and a storage medium capable of improving focus detection accuracy and acquiring a high-quality image.

FIG. 3 is a block diagram of an image processing unit in the first embodiment. It is explanatory drawing of the influence of shading. It is a block diagram of the imaging device in a 1st embodiment. It is the schematic of the unit pixel cell of the image pick-up element in 1st Embodiment. It is an array diagram of the color filter of the image sensor in the first embodiment. It is a related figure with the incident principal ray to the some photoelectric conversion part which shares a division pupil and a micro lens. It is explanatory drawing of the mode of the vignetting produced by the shift | offset | difference of an exit pupil and an aperture position. It is explanatory drawing of the influence of alignment with a micro lens and a photoelectric conversion part. It is explanatory drawing of the magnitude | size and shape of the exit pupil of several photoelectric conversion parts which share one microlens. It is explanatory drawing of the gradation change by the shading according to the change of a view angle or an image coordinate. It is explanatory drawing of the saturated pixel value in the case of shading correction | amendment. It is explanatory drawing of the influence of the saturated pixel in correlation calculation. It is explanatory drawing of the correlation calculation by the multipoint phase difference ranging in 2nd Embodiment. FIG. 6 is an explanatory diagram of suppression in consideration of neighboring pixels in the first embodiment. 3 is a flowchart illustrating a method for controlling the imaging apparatus according to the first embodiment. It is a block diagram of the imaging device in 3rd Embodiment. It is a block diagram of a shading correction unit and an image processing unit in the first embodiment. In 1st Embodiment, it is explanatory drawing of the relationship between the amount of parallax, and a suppression gain.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In each figure, the same members are denoted by the same reference numerals, and redundant description is omitted.

(First embodiment)
First, the imaging device according to the first embodiment of the present invention will be described with reference to FIG. FIG. 3 is a block diagram of the imaging apparatus 300 in the present embodiment. The imaging apparatus 300 includes an optical system 301, an imaging element 302, an image processing unit 303 (image processing apparatus), and a CPU 304. The optical system 301 forms (condenses) light emitted from the subject 100 on the image sensor 302. The optical system 301 includes, for example, a plurality of lenses, a mirror, a diaphragm mechanism, and a driving mechanism for focusing and zooming.

  The image sensor 302 includes a plurality of photoelectric conversion units (hereinafter “PD”) corresponding to one micro lens (hereinafter “ML”) (that is, sharing one ML). FIG. 4 is a schematic diagram of a unit pixel cell 401 of the image sensor 302 including a plurality of photoelectric conversion units sharing one ML. 4A shows a cross-sectional view of the unit pixel cell 401, and FIGS. 4B and 4C show front views when a PD sharing one ML is divided into two and four, respectively. ing. As shown in FIG. 4A, a unit pixel cell 401 configured by a pixel group corresponding to one ML includes a first divided PD 402, a second divided PD 403, an ML 404, a color filter 405, and And the wiring layer 406. FIGS. 4B and 4C show an arrangement example of the two-divided PDs 402 and 403 and the four-divided PDs 402, 403, 404, and 405 that share the ML 404 in the unit pixel cell 401. FIG. 4B and 4C, incident light passes through the ML 404, has spectral characteristics by the color filter 405, and is irradiated to each divided PD. The color filter 405 is arranged for each unit pixel cell at a constant cycle.

  In the following description, when an image sensor including two PDs sharing one ML is used as shown in FIG. 4B, these two PDs are respectively connected to the first photoelectric conversion unit (PD1). And a second photoelectric conversion unit (PD2). In addition, when using an imaging device including four PDs sharing one ML as shown in FIG. 4C, at least one of the four PDs is a first photoelectric conversion unit (PD1), four At least one other (remaining) PD is referred to as a second photoelectric conversion unit (PD2).

  FIG. 5 is an arrangement diagram of color filters in the present embodiment, and shows an example of two-divided PDs 402 and 403. In FIG. 5, the color filter 405 exemplifies a general Bayer arrangement in which a set of R, G, G, and B (2 × 2 unit pixel cells) is periodically arranged. The structure and characteristics of the image sensor 302 of the present embodiment are the same as those of a general image sensor except that the unit pixel cell has a divided structure.

  The ML 404 serves as a field lens for the main optical system (main lens) configured by the optical system 301. For this reason, the light flux from the exit pupil that reaches the split PD is limited, and the light flux that has passed through different areas of the exit pupil reaches each split PD. That is, the first photoelectric conversion unit and the second photoelectric conversion unit are configured to receive light beams that pass through different regions of the exit pupil of the photographing optical system. For this reason, there is a parallax between an image formed by collecting pixels of the same arrangement in the unit pixel cell, for example, pixels of only PD 402 from the entire light receiving region, and an image formed by collecting pixels of only PD 403. Arise. By using such parallax between images, focus detection and stereo image processing can be performed. An image having parallax between pixels is called a parallax image.

  In addition, it is possible to consider that the spatial coherency between the divided pupils is low under a photographing illumination condition in which light with low coherence such as general sunlight is used for illumination and a condition using a consumer optical element. For this reason, an image obtained by adding pixel values between the PDs in the unit pixel cell 401, for example, between the two divisions 402 and 403 or between the four divisions 402, 403, 404, and 405, is a conventional non-pupil division optical system. It is obtained as a value approximately equivalent to the image photographed in. By performing image processing including pre-processing and post-processing mounted on the image processing unit 303 described later or general image processing on the added image, the same as in the case of using a conventional image sensor An image can be obtained.

  In addition, the imaging element 302 outputs parallax image signals corresponding to the number of divisions corresponding to each PD sharing one ML. In many cases, the readout circuit is shared, and each parallax image signal is sequentially output by sequential readout by switching the switching circuit, for example. Depending on the imaging device, in order to save the occupied transmission band, four arithmetic operations are performed between two or more parallax images, for example, a signal obtained by adding two or more parallax images for each pixel and a non-addition image are output as a set. In the case of an addition image output device, an image signal equivalent to the case of outputting for each parallax image signal can be obtained by performing separation processing in an image processing unit described later and restoring each parallax image.

  The image processing unit 303 can be broadly classified into a pre-processing unit 3031 and a post-processing unit 3032. The pre-processing unit 3031 removes noise by phase double sampling (CDS) from the analog image signal photoelectrically converted by the image sensor 302, gain-up by auto gain control (AGC), black level correction, A / D conversion, Perform basic processing such as scratch correction. The preprocessing unit 3031 obtains an image signal obtained by converting a signal after various processing into a digital signal. Since the preprocessing unit 3031 mainly performs preprocessing for analog signals, the main part is also called AFE (analog front end). What is used in pair with a digital output sensor is called a DFE (digital front end).

  On the other hand, the post-processing unit 3032 performs image processing relating to picture creation and coding / compression processing for recording / transmission on the image signal converted into the digital signal by the pre-processing unit 3031. The post-processing unit 3032 performs, for example, processing such as Bayer array interpolation, linearization matrix processing, white balance adjustment, YCC conversion, color difference / tone / contrast correction, and edge enhancement. In recent years, added value improvement processing such as dynamic range expansion processing and super-resolution processing for generating a wide dynamic range image by combining a plurality of images is also classified as one of post-processing. By these processes, information of an output image formed on one sheet or a moving image is generated. Further, these post-processing are called DBE (digital back end) processing with respect to the AFE processing of the pre-processing unit 3031.

  The image information generated by the image processing unit 303 is temporarily stored in a work memory composed of a DRAM (not shown) or the like, or directly transmitted to an assumed subsequent processing unit. For example, the processing unit in the subsequent stage includes a recording unit made of a semiconductor memory, a display unit made up of a display such as a liquid crystal, and an I / F (interface) that can be connected to a wired cable such as a wireless LAN or USB An external input / output I / F can be mentioned.

  The CPU 304 is a control unit that controls the operation of the entire imaging apparatus. For example, the CPU 304 controls the focus, zoom, aperture, and the like of the optical system 301. Based on a command from the CPU 304, the lens group and the diaphragm included in the components of the optical system 301 are driven. That is, the CPU 304 always manages state information of camera parameters such as a focus distance, a focal distance, and an aperture value.

  In the case of having an image sensor 302 including a plurality of PDs for one ML, or in an image processing apparatus that receives the video signal, an image processing unit 303, mainly a preprocessing unit 3031, is a main element of the present embodiment. The pixel value shading correction process is performed. Further, the post-processing unit 3032 performs ornamental image generation based on distance measurement processing using a parallax image, which is an added value processing, and pixel addition. The distance measurement information obtained by the image processing unit 303 is transmitted to the CPU 304, and the CPU 304 performs focusing (focus control) of the optical system 301 based on this information. The camera parameters such as the subject distance after focusing, the aperture value at that time, and the zoom state are held in the CPU 304. When using a more accurate camera parameter for control, the state of the camera parameter is measured using an encoder (not shown) mounted on the unit of the optical system 301 and the measured state is recorded in the CPU 304.

  The definition of pre-processing and post-processing in the above-described apparatus configuration, the classification, grouping and order of image processing included in each processing unit are ambiguous due to the recent rapid development of image processing and the evolution of devices. It has become a thing. For example, the shading correction processing may be performed as part of the latter half of post-processing such as distance measurement or ornamental image generation processing. Conversely, pixel addition between parallax images may be positioned as basic image processing and may be implemented as the first half of preprocessing. In addition, in AFE, which has been the mainstream in the past, complicated processing such as shading correction has been performed in DBE because it is not good at it. However, since DFE is becoming mainstream, there is an influence such as being implemented as preprocessing due to commonality of processing. When the image sensor 302 includes an image processing circuit on-chip, pre-processing and post-processing, shading correction unique to the image sensor 302 having a plurality of PDs sharing one ML, distance measurement processing, and ornamental image generation are performed. All the pixel additions for this purpose may be performed within the image sensor 302. Further, in the case of the image sensor 302 that outputs a set of parallax images added with pixels and non-added images, separation processing between the added image and the non-added image before the shading correction and distance measurement processing of the present embodiment, for example, pixel unit A process of performing level four arithmetic operations and separating the parallax images is added.

  Thus, there are various apparatus and processing configurations applicable to the present embodiment, and the pixel value shading correction processing described below is not limited to the above-described apparatus and processing configuration and their order. For this reason, the shading correction processing of the pixel value, which is the main element of the present embodiment, will be described below under the conditions in which the difference in conditions such as the above-described trivial combinations of the outputs of the image sensor 302 and the processing order are omitted. .

  Next, an image processing unit 303 (image processing apparatus) that performs shading correction in the present embodiment will be described with reference to FIG. FIG. 1 is a block diagram of the image processing unit 303 in the present embodiment. The image processing unit 303 includes a shading correction unit 3033, an image processing unit 3034, and a distance measurement unit 3035 (focus detection unit), and is configured to be able to perform shading correction.

  As shown in FIG. 1, a plurality of parallax images (image signals) captured by an image sensor 302 having a plurality of PDs for one ML are input to the image processing unit 303. It is assumed that preprocessing such as A / D conversion for the image signal output from the image sensor 302 and separation processing in the case where a part of the image signal is an added image have been performed. The shading correction unit 3033 performs shading correction in consideration of the pupil division system, and outputs a group of parallax images composed of a plurality of sheets corrected for the influence of saturation. The image processing unit 3034 generates an ornamental image based on the added image obtained by the pixel addition process. An image processing unit 3034 that performs conversion into an ornamental image corresponds to the post-processing unit 3032 that performs Bayer array interpolation, encoding compression, and the like. The distance measuring unit 3035 performs distance measurement processing (focus detection processing) based on a plurality of corrected parallax images (corrected first signal and second signal). The plurality of parallax images are used for obtaining dense depth information, stored in a recording unit (not shown), or output from an external input / output I / F and used as input to external image processing means. Also good.

  Shading is caused by two types of effects. FIG. 2 is a diagram for explaining the influence of shading. FIG. 2A shows shading mainly due to an inevitable decrease in peripheral light amount even in a normal non-pupil division optical system. FIGS. 2B and 2C respectively show shading specific to the pupil division optical system that is the object of the present embodiment (in the case of two left and right divisions, and in the case of four divisions, up and down, left and right). Shading correction is performed to reduce both effects. In actual execution of processing, it is efficient to perform correction in consideration of the total of two effects.

  As described above, the pupil division optical system is configured by combining the optical system 301 with the ML in the unit pixel cell 401 of the image sensor 302 and the image sensor 302 having a plurality of photoelectric conversion units for one ML. can do. As can be observed from the tendency of FIG. 2B, the shading accompanying the pupil division causes a shading biased in the pupil division direction between a plurality of parallax images. Hereinafter, a specific shading factor generated by using the image sensor 302 having a plurality of PDs sharing the ML first will be described.

  FIG. 6 is a diagram illustrating a relationship between a divided pupil (a partial region obtained by dividing the exit pupil) and a principal ray incident on a plurality of photoelectric conversion units PD sharing one microlens (ML). FIG. 6A is an explanatory diagram of the principal ray incident on the unit pixel cell near the center of the image corresponding to all pupils, and FIG. 6B is an explanatory diagram of the incident principal ray incident on the unit pixel cell in the peripheral portion of the image. . Here, the image center is the intersection of the optical axis of the optical system 301 and the image plane formed on the image sensor 302.

  In FIG. 6A, the angles of chief rays incident from each divided pupil are substantially equal. On the other hand, as shown in FIG. 6B, the angle of the principal ray from each divided pupil that enters the unit pixel cell in the peripheral portion of the image is relatively greatly different based on a simple geometric relationship. As a result, even if a luminous flux having the same luminance or illuminance from the same point of the subject is projected between the pair of pixels sharing one ML due to the influence of COS4 power measurement, which is a general shading factor, the pixel value There will be a difference. This is the main cause of the unique shading that occurs in the split pupil optics. When the distance from the main lens to the PD is significantly different between paired pixels sharing one ML, the pupil size of the main optical system is similarly different, the unit pixel from the image center Similarly, when the distance of the cell 401 is extremely large, it becomes prominent. Here, the distance from the main lens to the PD indicates not a physical distance but a virtual distance when the lens group constituting the main lens is converted into one synthetic lens.

  In the case of an ideal state without vignetting, for example, a lens having a long focal length, the COS4 power measurement faithfully models a shading phenomenon peculiar to pupil division. On the other hand, in a short focal length lens, shading due to vignetting due to a lens barrel or a lens holding portion is likely to occur, which is often a major factor. Further, while the exit pupil moves due to zoom and focus control, when the aperture position does not move, shading due to vignetting is more likely to occur.

  FIG. 7 is a diagram for explaining a vignetting state having a bias caused by a shift between the exit pupil and the position of the diaphragm 700 (aperture position). FIG. 7A is a diagram illustrating a case where the exit pupil and the aperture position coincide with each other. Normally, the stop position in the optical system 301 is set to the position of the exit pupil where the fluctuation of the path of the peripheral ray is the smallest with respect to the fluctuation of the angle of view. For this reason, there is no great difference between the vignetting for the angle of view near the center of the image and the vignetting for the peripheral portion of the image. In FIG. 7A, the left circle is the exit pupil 701 viewed from the periphery of the image, and the right circle is the exit pupil 702 viewed from the center of the image. The exit pupils 701 and 702 have substantially the same shape and size.

  However, the exit pupil may be displaced in the optical axis direction with respect to the stop position included in the optical system 301 depending on the zoom or focus control state. FIG. 7B illustrates a state in which vignetting occurs in a form in which the exit pupil is decentered by the diaphragm moved to a shifted position with respect to the light beam reaching the periphery of the image sensor 302. In this case, specific shading occurs between the angle of view near the center of the image and the angle of view of the peripheral part. In FIG. 7B, the left circle is the exit pupil 703 viewed from the periphery of the image, and the right circle is the exit pupil 704 viewed from the center of the image. The pupil (exit pupil 703) viewed from the peripheral angle of view is reduced by being shielded by the diaphragm 700 installed at a position shifted from the exit pupil. In addition, when the composite principal point of the optical system 301 is located near the image sensor 302, the optical axis connecting the main lens of the optical system 301 and the ML may be greatly inclined in the unit pixel cell as it approaches the periphery.

  FIG. 8 is a diagram for explaining the influence of alignment between the microlens ML and the photoelectric conversion unit PD. A unit pixel cell having an angle of view at the periphery of the image as shown in FIG. When the alignment between the ML and the PD is appropriate, the light flux from the exit pupil EP can be efficiently received by each PD as shown in FIG. 8B. On the other hand, when the alignment between ML and PD is not properly performed, it is as shown in FIG. Here, assuming that the image pickup element 302 side has high telecentricity of the optical system 301, the alignment between ML and PD is substantially equal pitch. On the other hand, when the telecentricity of the optical system 301 is low, a light beam to be incident on another divided pixel enters the adjacent PC and crosstalk occurs, or a part of the light beam is generated as shown in FIG. Vignetting is caused by the pixel structure and the wiring layer 406, causing a relative decrease in the amount of light. This is because, similar to the vignetting caused by the lens barrel and the lens holding portion described above, the influence appears as a reduction in the exit pupil size seen from the cross-sectional area of each PD, and the influence becomes larger as the angle of view at the peripheral portion of the image.

  FIG. 9 is an explanatory diagram of the size and shape of the exit pupil of a plurality of photoelectric conversion units PD sharing one microlens ML. Each of the cases of 2 divisions (2 divided) and 4 divisions (4 divided) is shown. Show. As shown in FIG. 9, due to the above-described complex factors, the size and shape of the exit pupil of the PD change according to the position in the unit pixel cell. The lower left in FIG. 9 is a schematic diagram of the exit pupil viewed from a unit pixel cell having an angle of view near the center of the image. In the case of the vicinity of the center of the image, the exit pupil shapes expected for a plurality of PDs sharing one ML are substantially the same. When the subject has a uniform luminance or illuminance surface and the same object point is projected in focus, the pixel values are substantially the same.

  On the other hand, the upper right in FIG. 9 is a schematic diagram of an exit pupil viewed from a pixel cell having an angle of view around the image sensor 302. Since the amount of vignetting differs between the PDs, there is a difference in the shape and size of the exit pupil, and there is a difference in the obtained pixel values. However, the vignetting shape generated by the lens barrel and the lens holding portion becomes a complicated shape depending on the lens barrel shape and the like. In addition, the vignetting shape caused by the assumed shift of the telecentric characteristic between the optical system 301 and the image pickup element 302 changes in magnitude due to the shift of the relative characteristic and causes direction reversal. Therefore, the shape shown in FIG. 9 is only a schematic shape and is an approximate one that represents a change in relative size that affects the light amount difference between pixel values.

  As described above, due to the influence of the COS-fourth law and the influence of the difference in exit pupil size seen from each divided pupil, there is a difference in the amount of light received between PDs sharing shading and sharing one ML. Arise. As a result, even if a plane with uniform luminance or illuminance is photographed, spatially having directionality as shown in FIG. 2 (b) in the case of two divisions and in FIG. 2 (c) in the case of four divisions. Shading with uneven gradation difference occurs.

  FIG. 10 is an explanatory diagram of gradation change (shading characteristics) due to shading according to changes in the angle of view or image coordinates. FIG. 10 corresponds to an image profile when the subject has a uniform luminance or illuminance surface. In FIG. 10, the horizontal axis represents image coordinates (pixel position of the image) with respect to the pupil division direction passing through the center of the image. The vertical axis in FIG. 10 is a gradation value (or coefficient) obtained from the pupil division PD of each unit pixel cell of the image sensor 302. FIG. 10A shows the dimming characteristics caused by the optical system at the time of non-pupil division corresponding to an image sensor that does not perform pupil division or non-pupil division after pixel addition. Due to the general shading effect, the gradation value is dimmed as it goes to the peripheral angle of view.

  FIG. 10B is a distribution diagram of gradation changes due to shading caused by pupil division during pupil division. Here, a difference in gradation value distribution when isoluminous light is incident on two PDs (two-segment PDs) sharing one ML is shown. One parallax image to be paired is a parallax image 1 and the other is a parallax image 2. Which of the right and left in the unit pixel cell pixel structure is the parallax image 1 or the parallax image 2 varies depending on the condition of the shading factor described above. For this reason, the relationship of FIG.10 (b) is not exact | strict. The sampling values on the same two curves on the horizontal axis represent the gradation difference of the same image coordinates, in other words, the gradation difference of the PD sharing one ML, or the sensitivity difference from the viewpoint of the image sensor 302. The PD is affected by shading as it goes to the periphery of the image. For this reason, a large gradation difference occurs even when isoluminous light is incident on a plurality of PDs sharing one ML.

  FIG. 10C shows shading characteristics when FIGS. 10A and 10B due to the influence of shading are combined and considered simultaneously. The influence of the shading caused by the optical system at the time of non-pupil division in FIG. 10A and the shading caused by the pupil division in FIG. 10B uses at least the coordinate position in the pupil division direction as a variable with reference to the image center. Correction is performed by gain correction using a correction coefficient. Specifically, the variable of the coordinate position in the pupil division direction with reference to the center of the image is the horizontal coordinate in the case of pupil division in the horizontal direction, and in the case of pupil division in the horizontal and vertical direction. It is a variable that represents the image coordinates in the horizontal and vertical directions. FIG. 10D is an example of a shading correction parameter for the combined shading characteristic of FIG. Shading correction is performed by multiplying the correction parameter by the corresponding gradation value of the image coordinates as a variable. When the shading correction parameter has a coefficient in one or more expansion directions as shown in FIG. 10D, it is preferable to implement scaling or bit depth expansion simultaneously with gain correction by the coefficient. FIG. 10E shows the correction result. The effect of shading is corrected between parallax images and within each parallax image, and uniform sensitivity, in other words, equal tone characteristics are obtained.

  The above-described shading correction parameters relating to pupil division, for example, the correction coefficient using the horizontal coordinate as the variable and the correction coefficient using the horizontal / vertical coordinate as the variable are created by design parameters at the time of designing the imaging apparatus or by measurement using the imaging apparatus. The correction coefficient can also be created in a form including the above-described shading characteristics due to the optical system during non-pupil division.

  Specifically, the correction coefficient is based on an optical system 301 (imaging optical system), an imaging element 302, a lens barrel structure including an aperture, and camera parameters at the time of imaging such as zoom, focal length, and aperture value. It is obtained by optical calculation, for example, calculation by geometrical optical calculation. In practice, the function of the optical design CAD is often used. First, by using an optical system design CAD that designs the optical system 301, shading data including a gradation distribution when an isoillumination surface is imaged is calculated for each divided pupil by ray tracing. At this time, the data of the lens barrel and the lens holding unit is also used to calculate the influence of shading in consideration of vignetting caused by the lens barrel and the lens holding unit. As shown in FIG. 10 (c), the shading characteristic due to the optical system at the time of non-pupil division in FIG. 10 (a) and the shading characteristic due to the pupil division in FIG. 10 (b) can be obtained simultaneously. An efficient calculation can be realized at the time of correction. Then, this gradation distribution is calculated for each divided pupil by using a horizontal coordinate or a horizontal / vertical coordinate as a variable as a correction coefficient that makes an equal gradation based on the image center coordinates. As described above, FIG. 10D is a composite correction parameter, and is calculated as a parameter for correcting the shading characteristics of FIG.

  As the above-mentioned optical system design CAD, Synopsys Inc. (ORA) developed by CODEV and LightTools, Radiant ZEMAX LLC. Representative examples include products such as ZEMAX developed by the company. These design CADs have dedicated commands for calculating the effect of shading on the imaging surface based on ray tracing. In addition, when calculating shading characteristics in consideration of a more detailed vignetting effect caused by the wiring layer 406, particularly when the basic function is insufficient, a pixel cell is used by utilizing a macro function provided in each design software. This can be realized by including the following structural model in the calculation. The macro is input so that the structure of the unit pixel cell 401 shown in FIG. When the wave optical influence of the pixel cell structure is strictly calculated, the influence by shading is calculated by using a calculation output by software capable of structural analysis by a finite element method or the like using the cooperation function. The correction parameter may be created by measurement using an imaging device. The shading data is measured from the gradation value for each divided pupil of each unit pixel cell when the isoilluminance plane is imaged, and the correction coefficient that makes the gradation distribution equal to the image center coordinates is set to the horizontal coordinate or horizontal. You may calculate for every division pupil by making a perpendicular coordinate into a variable.

  Correction parameters obtained by calculation or measurement by design CAD in advance are stored in a recording unit such as a ROM (not shown) included in the image processing unit 303. Correction parameters may be prepared only for typical combinations of camera parameters, or may be prepared in a multi-dimensional table format that covers the domain of all parameter variables.

  When the dynamic range of the image sensor 302 is sufficiently wide, the number of pupil divisions is small, the scene luminance range is narrow, and the exposure conditions of the imaging device 300 are appropriate, the gradation value of the captured image is obtained without being almost saturated. It is done. For this reason, the gradation difference between parallax images is appropriately corrected by the shading correction of this embodiment, and phase difference ranging between parallax images can be realized with sufficient accuracy. On the other hand, when the dynamic range of the image sensor 302 is narrow, when the number of pupil divisions is large, when the luminance range of the scene is wide, or when the exposure condition of the imaging device 300 is inappropriately applied to a low-luminance portion, shooting is performed. In this case, saturation occurs in a high gradation portion in the image. When the shading correction of the present embodiment is performed on the saturated pixels, a pseudo gradation difference is generated between the corresponding pixels of the parallax image.

  FIG. 11 is an explanatory diagram of saturated pixel values at the time of shading correction, and shows an example of a two-division PD. FIG. 11A shows the output characteristics of the two photoelectric conversion elements PD1 and PD2 included in the photoelectric conversion unit and the combined output characteristics of the outputs of PD1 and PD2 when the saturation countermeasure of this embodiment is not performed. The combined output can be obtained by adding at least the signals of the photoelectric conversion elements. The composite output may be obtained by averaging. Further, additional processing such as amplification may be performed. For the sake of explanation, FIG. 11 shows a case where the sensitivity of PD1 is higher than that of PD2, that is, PD1 receives more light than PD2.

  When the incident light on the photoelectric conversion elements PD1 and PD2 is within the range 1101, the generated charge is larger in PD1 than in PD2. Since PD1 is not saturated, an appropriate output can be obtained by combining PD1 and PD2. However, when the incident light is within the range 1102, PD1 is saturated and PD2 is not saturated. In this case, only the PD 2 outputs a signal having linearity according to the incident light. For this reason, a tone difference occurs in the signal subjected to the shading correction within the range 1102. With respect to such a problem, in Patent Document 2 (Japanese Patent Laid-Open No. 63-276010) and the like, when at least one pixel among a plurality of PDs sharing one ML is a saturated pixel, there is a gap between the pixels. A countermeasure has been proposed in which the calculation is excluded from the correlation calculation target.

  FIG. 12 is an explanatory diagram of the influence of saturated pixels in the correlation calculation. FIG. 12A shows a profile in the correlation including the saturation region. For example, the absolute value summation of the differences is performed while shifting the parallax image 1 (1201) and the parallax image 2 (1202), and the correlation is calculated. FIG. 12A shows an example of correlation calculation when no shading correction is performed. Features near the boundary of saturated tone pixels have a positive effect on the correlation. However, in actuality, in the non-saturated region, a difference occurs in the pixel value even at the in-focus phase position due to the influence of shading, and the correlation calculation accuracy decreases.

  FIG. 12B is an example in which correlation calculation is performed by pixel replacement after shading correction. On the contrary, in the non-saturated region, the correlation calculation accuracy is improved by the effect of shading correction. On the other hand, if the shading characteristics of the saturated gradation pixel portion are different in the peripheral portion of the image, the gradation is affected by the shading correction even if it is a pair of pixels projected with an isoluminance or isoluminance object point.

  FIG. 12C shows a case where, when at least one pixel among a plurality of PDs sharing one ML is a saturated pixel, the calculation between the pixels is excluded from the correlation calculation target. A region surrounded by a dotted line is a region where at least one pixel is saturated. When the texture characteristics are low such as the profile characteristics shown in the figure, if the saturation region is excluded, the correlation calculation must be performed using only the minute features on the right side of the saturation of the profile in FIG. Tends to increase. For this reason, the reliability of the correlation calculation decreases. There is a high possibility that the minute feature is a false correspondence feature caused by illumination change or noise.

  Saturation also affects ornamental image generation by the sum of the gradation values of a plurality of PDs sharing one ML. In the range 1102 where PD1 is saturated, the composite output of FIG. 11A changes in tone depending on only the output of PD2. As a result, the synthesized output has a non-linear sensitivity characteristic with respect to the input called the knee characteristic from the point where PD2 is saturated. The gradation value acquired with such a knee characteristic is captured by an image sensor 302 having a color filter array as shown in FIG. 5 and processed as a color image by a subsequent image processing unit. It appears as a prominent phenomenon called color bending or color shading.

  Therefore, in the present embodiment, when there is at least one unsaturated pixel sharing one ML with respect to a saturated pixel, the shading characteristic value or the correction value is used depending on the gradation value of the unsaturated pixel sharing one ML. Thus, the gradation value when the saturated pixel value is not saturated is estimated. A saturated pixel is a pixel having a gradation value that exceeds a certain threshold with respect to the gradation value before shading correction. The threshold value is determined in consideration of noise inherent in the image sensor, signal fluctuation, and the like. In other words, when any PD sharing one ML is above a predetermined level, that is, when it can be regarded as saturated, the level at which saturation does not occur in a saturated pixel is determined based on the pixel level that has not reached the predetermined level. Estimate and correct sensitivity differences. In other words, the shading correction means has a predetermined level or higher when at least one of the imaging signals obtained from the plurality of photoelectric conversion units corresponding to the divided pupil images included in the same single pixel cell is equal to or higher than the predetermined level. Consider a pixel as saturated. Then, based on the level of the pixel that has not reached the predetermined level, the level at which saturation does not occur in the saturated pixel is estimated to correct the sensitivity difference.

  FIG. 11B is a diagram illustrating a saturation countermeasure processing method according to this embodiment. Based on the non-saturated low-brightness signal PD2, the original tone of the saturated PD1 is estimated. Estimation is performed on the assumption that subjects having the same luminance or equal illuminance are projected. For the estimation, shading characteristic information or correction information is used. The other can be calculated from one. Using the shading characteristic data of the parallax image 1 and the parallax image 2 having the same coordinates, and the tone value of the PD 2, the tone value before shading correction can be estimated by the following equation (1).

  However, in practice, it is only necessary to obtain a gradation value after shading correction. For this reason, equation (1) is described more simply as represented by equation (2) below.

  By performing such correction, it is possible to measure the gradation of PD1, which is saturated immediately, as shown in FIG. 11A, with a wider dynamic range. In order to correct each parallax image, in the input gradation region in the range 1102, the estimation is reflected also in the synthesized output image, and an estimated synthesis characteristic maintaining a linear characteristic as shown in FIG. 11B can be realized. The thick line in the figure is a characteristic straight line obtained by correction.

  However, when some of the divided pupil pixels sharing one ML are saturated pixels, the remaining non-saturated pixel signal level is less than the minimum value corresponding to the saturated pixel signal level. There is a high possibility that another object point is not projected. For this reason, it cannot estimate correctly with the said method. FIG. 11C shows an example of that case (in the case of two divisions). Consider a case where the high luminance side signal level 1104 is already saturated, whereas the unsaturated low luminance side signal level 1105 is less than the minimum gradation threshold 1103 corresponding to the saturation estimated from the shading characteristics. In this case, in a normal scene, there is a high possibility that the corresponding pixel has a mapping of a subject having a different out-of-focus portion, that is, parallax. For this reason, when the parallax is generated and one of the divided pupil pixels is a saturated pixel, the value estimated by the above method is different from the original signal value. For this reason, coloring different from the original color occurs in the combined output signal.

  Therefore, when a part of the divided pupil pixels is saturated, the imaging apparatus 300 according to the present embodiment suppresses coloring using the subsequent saturation suppression unit based on the amount of parallax generated in the pixel. Process. Next, with reference to FIG. 17 and FIG. 18, processing for suppressing coloring by the saturation suppressing unit (gain setting unit) will be described.

  FIG. 17 is a block diagram of a shading correction unit and an image processing unit that constitute a part of the image processing apparatus of the present embodiment. In FIG. 17, a shading correction unit 1700 and an image processing unit 1710 correspond to the shading correction unit 3033 and the image processing unit 3034 in FIG. 1, respectively. FIG. 18A is a diagram illustrating the relationship between the sensitivity difference characteristics of the pupil-divided imaging system, and FIG. 18B is an explanatory diagram regarding the determination of the suppression gain. The case of two divisions will be described as an example.

  In FIG. 17, reference numeral 1701 denotes saturation determination means (determination means). Saturation determination means 1701 is configured such that a signal (first signal) obtained from the first photoelectric conversion unit (PD1) and a signal (second signal) obtained from the second photoelectric conversion unit (PD2) are predetermined. It is determined whether the level (saturation level) has been reached. That is, the saturation determination unit 1701 determines whether or not the signal level of the image signal (pupil division signal) obtained from the pupil division imaging system (imaging element 302) shown in FIG. 1 is saturated. Then, the saturation determination unit 1701 sends the determination result (saturation determination information) to a saturation correction value generation unit described later. Reference numeral 1702 denotes a saturation correction value generation unit. The saturation correction value generation unit 1702 generates a saturation correction value for correcting the image signal based on the determination result (saturation determination information) obtained from the saturation determination unit 1701.

  Reference numeral 1703 denotes saturation correction means (correction means). The saturation correction unit 1703 corrects the first signal or the second signal based on information from the saturation determination unit 1701. That is, the saturation correction unit 1703 corrects the image signal (pupil division signal) using the saturation correction value generated by the saturation correction value generation unit 1702. Reference numeral 1704 denotes a database. The database 1704 records information related to the sensitivity difference caused by the pupil division imaging system of each PD that constitutes the same pixel portion of the parallax image read out by dividing the pupil, which is necessary for generating the saturation correction value. ing. This information is used when the saturation correction value generation unit 1702 generates a saturation correction value.

  Reference numeral 1711 denotes image addition means. The image addition unit 1711 adds the pupil division signals (corrected pupil division signals) output from the shading correction unit 1700 and outputs one pixel signal. Reference numeral 1712 denotes image signal processing means. The image signal processing unit 1712 performs various processes such as Bayer array interpolation, linearization matrix processing, white balance adjustment, YUV conversion, color difference / gradation / contrast correction, and edge enhancement, and outputs a YUV signal (Y: Luminance signal, UV: color difference signal).

  Reference numeral 1713 denotes a parallax amount calculating means (calculating means). The parallax amount calculation means 1713 calculates the parallax amount between the first photoelectric conversion unit (PD1) and the second photoelectric conversion unit (PD2). That is, the parallax amount calculation unit 1713 calculates the parallax amount based on signal level information (pupil division signal) obtained from the pupil division imaging system (imaging element 302). Reference numeral 1714 denotes a suppression gain calculation means. The suppression gain calculation unit 1714 calculates a gain (suppression gain) based on the parallax amount calculated by the parallax amount calculation unit 1713.

  Reference numeral 1715 denotes saturation suppression means (gain setting means). The saturation suppression unit 1715 is a signal (a color such as a UV signal) obtained from the corrected first signal and second signal based on the information from the saturation determination unit 1701 and the information from the parallax amount calculation unit 1713. The level of the signal containing information). That is, the saturation suppression unit 1715 converts the suppression gain calculated by the suppression gain calculation unit 1714 according to the amount of parallax to the UV signal (color difference signal or color signal) with respect to the YUV signal output from the image signal processing unit 1712. Multiply and output.

  Next, the saturation determination unit 1701, the parallax amount calculation unit 1713, the suppression gain calculation unit 1714, and the saturation suppression unit 1715 will be described in detail. First, the saturation determination unit 1701 will be described. FIG. 18 (a) shows the sensitivity difference characteristics in two divisions, and the outputs of PD1 and PD2 are Val1 and Val2, respectively. FIG. 18A shows an example in which the sensitivity of PD1 is higher than the sensitivity of PD2. If the amount of parallax is 0, PD1 is saturated before PD2, and the value of Val1 at that time is Th1. To do. Further, when the parallax amount is 0, the value of Val2 when the value of Val1 has just reached Th1 is Th2. That is, the difference between Th1 and Th2 indicates the sensitivity difference characteristic between PD1 and PD2. When Val1 reaches Th1 and Val2 is equal to or greater than Th2, some of the divided pupil pixels are saturated. In addition, since the sensitivity difference characteristic changes according to the image height position, the sensitivity of PD2 may be higher than that of PD1. In this case, when PD2 is saturated before PD1, the value of Val2 at that time is Th2 and the value of Val1 is Th1, when Val2 reaches Th2 and Val1 is equal to or greater than Th1, the divided pupil pixels Some pixels are saturated.

  Next, the parallax amount calculation unit 1713 will be described. In FIG. 18A, since the output of PD1 is the same value as Th1, PD1 is saturated and the input signal is within a range 1802. In this case, since the output of PD2 is one on the sensitivity difference characteristic line, it should have a value between Th2 and Th1. However, when the output of PD2 is less than Th2, it can be determined that there is a high possibility that a signal that does not conform to the sensitivity difference characteristic is incident on PD1 and PD2, that is, a parallax has occurred. Further, the amount of parallax indicating how much parallax is generated can be defined as how far away from the saturation level, that is, the difference amount (Th2-Val2). A positive value indicates a larger parallax, and a negative value indicates that no parallax has occurred. In addition, when the output of PD2 is a value between Th2 and Th1, it cannot be said that there is no possibility of occurrence of parallax. It is defined that parallax occurs. However, the present embodiment is not limited to this, and it is possible to make a determination in consideration of the possibility.

  Next, the suppression gain calculation unit 1714 will be described. FIG. 18B is a diagram illustrating the relationship between the parallax amount (Th2-Val2) on the horizontal axis and the suppression gain (gain) on the vertical axis. Th2 is the saturation level of PD2, and is determined according to the image height position. When the output (Val2) of PD2 is smaller than Th2, that is, when (Th2-Val2) is positive, since there is a high possibility that parallax has occurred, control is performed to perform suppression. More specifically, it is assumed that the larger the amount of parallax, the greater the coloring, and the suppression gain is controlled to approach 0 as (Th2-Val2) increases. On the other hand, when (Th2-Val2) is negative, since the possibility that parallax has occurred is low, the suppression gain is set to 1. That is, in this case, control is performed so that no processing is performed. Further, as described above, Th2 changes according to the image height position. For this reason, even if Val2 is the same, the suppression gain changes depending on the image height position. In addition, in the situation where parallax occurs, it is considered that coloring due to erroneous estimation always occurs, and in addition to changing the suppression gain according to the amount of parallax, if (Th2-Val2) is positive, the gain is set to 0 and forced Alternatively, saturation may be suppressed.

  Next, the saturation suppression unit 1715 will be described. The saturation suppression unit 1715 suppresses the UV signal that is a color difference signal based on the suppression gain (gain) calculated by the suppression gain calculation unit 1714 for the YUV signal output from the image signal processing unit 1712. Multiply the gain and output.

  Note that each of the above-described processes is performed in units of pixels. However, if the pixels for which the suppression process is performed and the pixels for which the suppression process is not performed are consecutive, an unnatural image may be generated. For this reason, suppression processing may be performed in consideration of surrounding pixels. FIG. 14 is an explanatory diagram of suppression in consideration of surrounding pixels. In the N × N region of FIG. 14, if even one pixel is saturated and parallax occurs, the suppression process is performed on all the pixels in the N × N region. Also good. At this time, the suppression gain uses a minimum value, a maximum value, an average value, or the like of gain values calculated for each pixel in the N × N region. As described above, the saturation suppression unit 1715 generates parallax in a plurality of surrounding photoelectric conversion units even when neither the first signal (PD1) nor the second signal (PD2) has reached a predetermined level. If so, the level of the signal including color information can be reduced.

  Next, with reference to FIG. 15, a control method of the imaging apparatus in the present embodiment will be described. FIG. 15 is a flowchart illustrating a method for controlling the imaging apparatus. Here, as in FIG. 17, a case of a two-segment PD will be described as an example. Each step in FIG. 15 is mainly performed by each element of the shading correction unit 3033 and the image processing unit 3034 of the image processing unit 303.

  First, in step S1501, the saturation determination unit 1701 performs saturation determination on the photoelectric conversion elements PD1 and PD2. In step S1502, the saturation determination unit 1701 determines whether or not the pixel values (Val1, Val2) of PD1 and PD2 are both saturated. If both are saturated, the process proceeds to step S1504, and the saturation correction unit 1703 does not correct the pupil division signal. Then, control goes to a step S1508. On the other hand, if at least one of PD1 and PD2 is not saturated in step S1502, the process proceeds to step S1503.

  In step S1503, the saturation determination unit 1701 determines whether or not the pixel values (Val1, Val2) of PD1 and PD2 are both saturated. If neither is saturated, the process proceeds to step S1505, and the saturation correction unit 1703 performs normal shading correction, and then proceeds to step S1508. On the other hand, if only one of PD1 and PD2 is saturated in step S1503, the process proceeds to step S1506.

  When PD1 is saturated in step S1506, the saturation correction unit 1703 estimates the pixel value of PD1 that is saturated from the pixel value of PD2 that is not saturated. In step S1507, the parallax amount calculating unit 1713 calculates the parallax amount (Th2-Val2), and the process proceeds to step S1508.

  In step S1508, the image addition unit 1711 and the image signal processing unit 1712 perform various types of image processing such as addition of pupil division pixels and YCC conversion on the output of each branch. In step S1509, the suppression gain calculation unit 1714 determines (calculates) a suppression gain according to the amount of parallax. Here, when both PD1 and PD2 are saturated or both are not saturated, the suppression gain is set to 1.0 (that is, not suppressed). When one pixel is saturated (in this embodiment, only PD1 is saturated), the suppression gain calculation unit 1714 calculates the suppression gain according to the parallax amount calculated in step S1507. Subsequently, in step S1510, the saturation suppression unit 1715 performs a process of suppressing the saturation on the output signal (UV signal) of the image signal processing unit 1712 using the suppression gain determined in step S1509. .

  Thus, in this embodiment, the saturation suppression unit 1715 (gain setting unit) is based on the information from the saturation determination unit 1701 and the information from the parallax amount calculation unit 1713, and the corrected first signal and second The level of the signal including color information obtained from the above signal is reduced. Preferably, the saturation suppression unit 1715 reduces the level of the signal including color information based on the amount of parallax when one of the first signal and the second signal has reached a predetermined level. Preferably, the signal including color information is a color signal or a color difference signal (UV signal) obtained from an addition signal (output signal of the image addition means 1711) of the corrected first signal and the second signal. is there. More preferably, the saturation suppression unit 1715 multiplies this signal by a gain so that the level of the signal including color information decreases as the amount of parallax increases.

  Preferably, the saturation correction unit 1703 has a sensitivity between the first signal and the second signal generated due to the incident angle of the principal ray of the light beam incident on the first photoelectric conversion unit and the second photoelectric conversion unit. Correct the difference. More preferably, when one of the first signal and the second signal has reached a predetermined level, the saturation correction unit 1703 determines the first signal and the second signal based on the other of the first signal and the second signal. The original level of one of the second signals is estimated and the sensitivity difference is corrected.

  Preferably, when the other level of the first signal and the second signal is smaller than the second level (Th2) determined based on the sensitivity difference, the parallax amount calculating unit 1713 generates a parallax. It is determined that More preferably, the parallax amount is a difference (Th2−Val2) between the other level of the first signal and the second signal and the second level. Preferably, the saturation determination unit 1701 determines that the first signal or the second signal has reached a saturation level when the first signal or the second signal has reached a predetermined level.

  According to the present embodiment, when any one of the pupil-divided pixels is saturated, it is possible to suppress (reduce) coloring caused by the occurrence of parallax when estimating from a pixel that is not saturated. it can.

(Second Embodiment)
Next, a second embodiment of the present invention will be described. The parallax amount calculation means 1713 of the present embodiment calculates the parallax amount using the distance measurement information. FIG. 13 is a diagram for explaining correlation calculation in multipoint phase difference ranging. FIG. 13A shows a subject that appears in one parallax image included in the parallax image group. Assume that the lower-side subject A is a distant subject, the center subject B is a focus subject, and the upper subject C is a foreground subject. FIG. 13B is a diagram illustrating a relationship between a distance measurement point, a distance measurement area, and pixels used for correlation of distance measurement. A relatively large black circle is a distance measuring point. The distance measuring point is a reference point of distance measuring information, and is located at the approximate center of the distance measuring area from the characteristics of the correlation calculation based on the pixel value. The distance measurement area (multi-point distance measurement area) is a group area of pixels related to distance measurement of the distance measurement points, and is an area surrounded by a thick-line square. A small black circle in the distance measurement area represents a pixel used for correlation calculation. The pixels used for distance measurement may be acquired or thinned out, for example, to improve calculation efficiency. Alternatively, pixel values corresponding to the same pupil of different color filters in the vicinity may be weighted and added to obtain luminance pixel information.

  In this manner, the correlation calculation between the left eye image and the right eye image is performed for each distance measurement area, and the pixel shift amount as shown in FIG. 13C can be calculated. It can be determined that a pixel included in a region where the amount of deviation is larger than a predetermined threshold is highly likely to have parallax. Therefore, it is also possible to determine a suppression gain according to the amount of parallax using the amount of deviation as the amount of parallax. Since the amount of deviation is determined when the image capturing is completed, a suppression gain is applied to the next captured image.

  As described above, in this embodiment, the parallax amount calculation unit 1713 calculates the parallax amount based on the distance measurement information obtained from the first photoelectric conversion unit (PD1) and the second photoelectric conversion unit (PD2). Preferably, the distance measurement information is a defocus amount (deviation amount) calculated based on the first signal and the second signal.

(Third embodiment)
Next, a third embodiment of the present invention will be described. FIG. 16 is a block diagram of the imaging apparatus 300a in the present embodiment. The imaging device 300a of the present embodiment is different from the imaging device 300 of the first embodiment in that it includes a database unit 305. The database unit 305 holds information used by the saturation correction unit 1703 (information regarding correction such as shading characteristics, correction values, and correction parameters).

  In the first embodiment, shading characteristics or correction parameters are stored in a recording unit such as a ROM (not shown) included in the image processing unit 303. However, for example, when recording is performed with fine sampling with respect to camera parameters such as a focus distance, zoom, and aperture setting value for the purpose of increasing accuracy, the recording amount of characteristics and correction parameters has a property of increasing linearly. This is because the shading characteristics and the apparent size of the exit pupil depend on the incident angle of the light beam to a plurality of PDs sharing the ML, that is, the geometrical positional relationship of the principal point position of the optical system with respect to the pixel unit and the size of the exit pupil. This is because it depends on (state of camera parameters such as focal length and aperture).

  Also, the characteristics and correction parameters become very large when the zoom range is wide. Further, when the optical system 301 is an interchangeable lens, it is necessary to take measures such as recording characteristics and correction parameters corresponding to the interchangeable lens in advance, updating them later, or obtaining them from a network or the like when necessary. The database unit 305 is a processing unit that performs such countermeasures. For this reason, the database unit 305 is configured by a ROM or a RAM. The database unit 305 is a means that can update data via an external I / F (not shown), for example, an SD card slot, or obtain necessary data via a network such as a wireless LAN. The CPU 304 inputs the camera parameters at that time of the imaging apparatus 300a to the database unit 305, and acquires shading characteristics or correction values that match the shooting state. Then, the CPU 304 transmits the acquired shading characteristics or correction value data to the image processing unit 303. The image processing unit 303 uses the shading characteristics or correction value data received from the CPU 304 for shading correction.

  According to the present embodiment, more accurate correction can be realized with low resources. In addition, it is possible to facilitate the correspondence of the interchangeable lens.

(Other embodiments)
The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or apparatus reads the program. To be executed. In this case, a computer-executable program describing the procedure of the imaging apparatus control method and a storage medium storing the program constitute the present invention.

  According to each embodiment, it is possible to provide an image processing device, an imaging device, an image processing method, a program, and a storage medium that can improve focus detection accuracy and acquire a high-quality image.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

  For example, in each embodiment, the case of two photoelectric conversion units (2-division PD) or four photoelectric conversion units (4-division PD) is described as a plurality of photoelectric conversion units sharing one microlens. However, it is not limited to these. Other division numbers may be used.

303 Image processing unit (image processing apparatus)
1701 Saturation determination means (determination means)
1703 Saturation correction means (correction means)
1713 Parallax amount calculation means (calculation means)
1715 Saturation suppression means (gain setting means)

Claims (17)

An image processing apparatus that processes an image signal output from an imaging device including a first photoelectric conversion unit and a second photoelectric conversion unit sharing one microlens,
Determining means for determining whether or not each of the first signal obtained from the first photoelectric conversion unit and the second signal obtained from the second photoelectric conversion unit has reached a predetermined level;
Correction means for correcting the first signal or the second signal based on information from the determination means;
Calculating means for calculating a parallax amount between the first photoelectric conversion unit and the second photoelectric conversion unit;
Gain setting means for reducing the level of the signal including color information obtained from the corrected first signal and second signal based on the information from the determination means and the information from the calculation means. And
The gain setting means reduces the level of the signal including the color information based on the amount of parallax when one of the first signal and the second signal has reached the predetermined level. A featured image processing apparatus. The image processing apparatus according to claim 1, wherein the signal including color information is a color difference signal obtained from an addition signal of the corrected first signal and the second signal. Said gain setting means, as the level of the signal including the color information about the amount of parallax is large is small, the image according to claim 1 or 2, characterized in that multiplying the gain with respect to the signal Processing equipment. The correction means is a sensitivity between the first signal and the second signal, which is generated due to an incident angle of a principal ray of a light beam incident on the first photoelectric conversion unit and the second photoelectric conversion unit. the image processing apparatus according to any one of claims 1 to 3, characterized in that to correct the difference. When one of the first signal and the second signal has reached the predetermined level, the correcting means determines the first signal based on the other of the first signal and the second signal. The image processing apparatus according to claim 4 , wherein the original level of one of the second signals is estimated to correct the sensitivity difference. The calculating means determines that a parallax has occurred when the other level of the first signal and the second signal is smaller than a second level determined based on the sensitivity difference. the image processing apparatus according to claim 4 or 5, characterized in. The image processing apparatus according to claim 6 , wherein the amount of parallax is a difference between the other level of the first signal and the second signal and the second level. The calculating means is any one of claims 1 to 7, characterized in that calculating the parallax amount on the basis of the first photoelectric conversion unit and distance measurement information obtained from the second photoelectric conversion unit The image processing apparatus according to item. If the parallax occurs in a plurality of surrounding photoelectric conversion units even if neither of the first signal and the second signal has reached the predetermined level, the gain setting means may change the color information. the image processing apparatus according to any one of claims 1 to 8, wherein reducing the level of a signal including. An image pickup device including a first photoelectric conversion unit and a second photoelectric conversion unit sharing one microlens;
Determining means for determining whether or not each of the first signal obtained from the first photoelectric conversion unit and the second signal obtained from the second photoelectric conversion unit has reached a predetermined level;
Correction means for correcting the first signal or the second signal based on information from the determination means;
Calculating means for calculating a parallax amount between the first photoelectric conversion unit and the second photoelectric conversion unit;
Gain setting means for reducing the level of the signal including color information obtained from the corrected first signal and second signal based on the information from the determination means and the information from the calculation means. And
The gain setting means reduces the level of the signal including the color information based on the amount of parallax when one of the first signal and the second signal has reached the predetermined level. An imaging device that is characterized. It said first photoelectric conversion unit and the second photoelectric conversion unit, to claim 10, characterized in that it is configured to receive light beams passing through different areas of the exit pupil of the photographing optical system The imaging device described. An adder for adding the first signal and the second signal;
The imaging apparatus according to claim 10 or 11 , wherein the signal including the color information is generated based on an addition signal added by the addition unit. The imaging apparatus according to any one of claims 10 to 12, further comprising a distance measuring unit for performing distance measurement processing based on the first signal and the second signal after the correction. The imaging apparatus according to any one of claims 10 to 13, further comprising a database unit that holds information used by the correction means. An image processing method for processing an image signal output from an imaging device including a first photoelectric conversion unit and a second photoelectric conversion unit sharing one microlens,
A determination step of determining whether or not each of the first signal obtained from the first photoelectric conversion unit and the second signal obtained from the second photoelectric conversion unit has reached a predetermined level;
A correction step of correcting the first signal or the second signal based on the information obtained in the determination step;
A calculation step of calculating a parallax amount between the first photoelectric conversion unit and the second photoelectric conversion unit;
Based on the information obtained in the determination step and the information obtained in the calculation step, the level of the signal including color information obtained from the corrected first signal and second signal is reduced. a setting step of setting the gain, the possess to,
In the setting step, when one of the first signal and the second signal reaches the predetermined level, the level of the signal including the color information is reduced based on the parallax amount. An image processing method. A program configured to cause a computer to execute the image processing method according to claim 15 . A storage medium storing the program according to claim 16 .