JP2015076754A5

JP2015076754A5 -

Info

Publication number: JP2015076754A5
Application number: JP2013212284A
Authority: JP
Filing date: 2013-10-09
Publication date: 2016-11-17
Anticipated expiration: 2033-10-09

Description

前記の目的を達成するために、本発明の第１の態様の画像処理装置は、撮像画素及び位相差検出画素を有する撮像素子からの画素出力を処理する画像処理装置であって、複数の前記画素出力を混合する画素混合部と、前記混合された画素出力毎の前記位相差検出画素の開口向き毎の画素出力の混入数及び前記撮像画素の画素出力の混入数の何れか１つ以上の情報を用いて、前記撮像画素の出力及び前記位相差検出画素の出力が混合された画素出力による画質劣化がないように前記混合された画素出力を補正する画像処理部とを具備することを特徴とする。 In order to achieve the above object, an image processing apparatus according to a first aspect of the present invention is an image processing apparatus that processes a pixel output from an image pickup device having an image pickup pixel and a phase difference detection pixel. Any one or more of a pixel mixing unit that mixes pixel outputs, a pixel output mixing number for each opening direction of the phase difference detection pixel for each mixed pixel output, and a pixel output mixing number for the imaging pixel And an image processing unit that corrects the mixed pixel output using information so that there is no deterioration in image quality due to the pixel output in which the output of the imaging pixel and the output of the phase difference detection pixel are mixed. And

前記の目的を達成するために、本発明の第２の態様の画像処理方法は、撮像画素及び位相差検出画素を有する撮像素子からの画素出力を処理する画像処理方法であって、複数の前記画素出力を混合し、前記混合された画素出力毎の前記位相差検出画素の開口向き毎の画素出力の混入数及び前記撮像画素の画素出力の混入数の何れか１つ以上の情報を用いて、前記撮像画素の出力及び前記位相差検出画素の出力が混合された画素出力による画質劣化がないように画素出力を補正することを特徴とする。 In order to achieve the above object, an image processing method according to a second aspect of the present invention is an image processing method for processing a pixel output from an image pickup device having an image pickup pixel and a phase difference detection pixel. The pixel outputs are mixed, and information on one or more of the number of pixel outputs mixed in the direction of opening of the phase difference detection pixel and the number of pixel outputs mixed in the imaging pixel for each of the mixed pixel outputs is used. The pixel output is corrected so that there is no deterioration in image quality due to a pixel output obtained by mixing the output of the imaging pixel and the output of the phase difference detection pixel .

前記の目的を達成するために、本発明の第３の態様の画像処理プログラムは、撮像画素及び位相差検出画素を有する撮像素子からの画素出力を処理する画像処理プログラムであって、複数の前記画素出力を混合する機能と、前記混合された画素出力毎の前記位相差検出画素の開口向き毎の画素出力の混入数及び前記撮像画素の画素出力の混入数の何れか１つ以上の情報を用いて、前記撮像画素の出力及び前記位相差検出画素の出力が混合された画素出力による画質劣化がないように画素出力を補正する機能とをコンピュータに実現させる。 In order to achieve the above object, an image processing program according to a third aspect of the present invention is an image processing program for processing a pixel output from an image pickup device having an image pickup pixel and a phase difference detection pixel. Information on at least one of the function of mixing pixel outputs, the number of pixel outputs mixed for each opening direction of the phase difference detection pixel for each mixed pixel output, and the number of pixel outputs mixed for the imaging pixel And causing the computer to realize a function of correcting the pixel output so that there is no deterioration in image quality due to the pixel output in which the output of the imaging pixel and the output of the phase difference detection pixel are mixed .

図２を用いて撮像素子２１の構成について説明する。図２は、撮像素子２１の画素配列の例を示した図である。図２は、ベイヤ配列の例であるが、カラーフィルタの配列はベイヤ配列に限るものではなく、種々の配列が適用され得る。 The configuration of the image sensor 21 will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of a pixel array of the image sensor 21 . FIG. 2 shows an example of the Bayer arrangement, but the arrangement of the color filters is not limited to the Bayer arrangement, and various arrangements can be applied.

比率計算処理部３１２１は、位相差検出画素の画素出力とその周辺の撮像画素の画素出力との比率を計算することにより、各位相差検出画素の画素出力をゲイン補正するための遮光量を算出する。遮光率算出処理部３１２２は、撮像制御回路２３による読み出しモード及び撮像素子２１における位相差検出画素の配置に応じた開口向きの異なる位相差検出画素の画素出力の混入数に従って、位相差検出画素の開口向き毎の遮光率（又は採光率）を算出する。遮光率の算出の詳細については後で説明する。 The ratio calculation processing unit 3121 calculates a light shielding amount for gain correction of the pixel output of each phase difference detection pixel by calculating the ratio between the pixel output of the phase difference detection pixel and the pixel output of the surrounding imaging pixels. . The shading rate calculation processing unit 3122 performs phase difference detection pixel detection according to the read mode by the imaging control circuit 23 and the number of mixed pixel outputs of the phase difference detection pixels having different opening directions according to the arrangement of the phase difference detection pixels in the image sensor 21. The shading rate (or lighting rate) for each opening direction is calculated. Details of the calculation of the shading rate will be described later.

画素データの選択後、画像処理部３１のＷＢ補正処理部３１１は、画素データに対してホワイトバランス補正処理を施す（ステップＳ１０４）。ここで、ホワイトバランス補正処理は、ループ処理の外で行われてもよい。続いて、ゲイン量推定部３１２の比率計算処理部３１２１は、比率計算処理を行う（ステップＳ１０５）。比率計算処理においては、現在の比率計算処理の対象となる位相差検出画素（画素混合が行われている場合には位相差検出画素の画素出力を含む画素）の画素出力とこの位相差検出画素の周辺の同色の撮像画素（画素混合が行われている場合には位相差検出画素の画素出力を含まない画素）の画素出力との比が算出される。例えば、撮像画素の画素出力をＮとし、位相差検出画素（例えば右開口位相差検出画素）の画素出力をｒとしたとき、画素出力の比Ｄｉｆ＿ｐｉは、以下の（式１）に従って算出される。
Ｄｉｆ＿ｐｉ＝ｒ/Ｎ（式１） After selecting the pixel data, the WB correction processing unit 311 of the image processing unit 31 performs white balance correction processing on the pixel data (step S104). Here, the white balance correction process may be performed outside the loop process. Subsequently, the ratio calculation processing unit 3121 of the gain amount estimation unit 312 performs ratio calculation processing (step S105). In the ratio calculation process, the pixel output of the phase difference detection pixel (the pixel including the pixel output of the phase difference detection pixel when pixel mixing is performed) and the phase difference detection pixel that are the targets of the current ratio calculation process The pixel output ratio of the image pickup pixels of the same color around the pixel (pixels that do not include the pixel output of the phase difference detection pixel when pixel mixing is performed) is calculated. For example, when the pixel output of the imaging pixel is N and the pixel output of the phase difference detection pixel (for example, the right aperture phase difference detection pixel) is r, the pixel output ratio Dif_pi is calculated according to the following (Equation 1). .
Dif_pi = r / N (Formula 1)

位相差検出画素の混入数が１個である画素については、ステップＳ１０５で計算された比率が遮光率として用いられる。例えば、２×２画素混合読み出しの場合、位相差検出画素の画素出力が混入される場合には、必ず１個の位相検出画素の画素出力が混入される。したがって、位相差検出画素の画素出力が混入されるすべての画素についてステップＳ１０５で計算された比率を遮光率として利用することができる。例えば、図７（ｂ）の画素ｒ１の遮光率は、画素ｒ１の画素出力をｒ１とし、画素ｒ１の周辺の同色画素Ｎの画素出力をＮとしたとき、以下の（式２）に従って算出される。画素ｌ１についても同様にして遮光率を算出することができる。さらに、３×３画素混合読み出しの場合、画素ｒ１と画素ｌ１については（式１）と同様に遮光率を算出することができる。
画素ｒ１の遮光率＝ｒ１／Ｎ（式２） For pixels with one phase difference detection pixel, the ratio calculated in step S105 is used as the light shielding rate. For example, in the case of 2 × 2 pixel mixed readout, when the pixel output of the phase difference detection pixel is mixed, the pixel output of one phase detection pixel is always mixed. Therefore, the ratio calculated in step S105 for all the pixels into which the pixel outputs of the phase difference detection pixels are mixed can be used as the light shielding rate. For example, the light blocking ratio of the pixels r 1 in FIG. 7 (b), when the pixel output of the pixel r 1 and r 1, the pixel output of the same color pixel N around the pixel r 1 was N, the following (Equation 2 ). The shading rate can be calculated in the same manner for the pixel 11 as well. Further, in the case of 3 × 3 pixel mixed readout, the shading rate can be calculated for the pixel r1 and the pixel l1 as in (Expression 1 ).
Shading rate of pixel r 1 = r1 / N (formula 2)

（式２）から画素ｌ２の遮光率は、以下の（式３）で示すように算出される。
画素ｌ２の遮光率＝ｌ２／Ｎ１（式３）
ここで、（１−遮光率）で計算される値を採光率と定義すると、画素ｌ２の採光率は、
以下の（式４）で示すように算出される。
画素ｌ２の採光率＝１−画素ｌ２の遮光率（式４）
光束の入射のし難さを示す遮光率に対し、採光率は光束の入射のし易さを示す。ところで、（式４）は２個の位相差検出画素が混入された状態の採光率である。これから位相差検出画素の混入数が１個だったとしたときの採光率は、２個の位相差検出画素が混入された状態の採光率の平均値として算出される。したがって、１個の画素ｌの採光率は、以下の（式５）で示すように算出される。
画素ｌの採光率＝画素ｌ２の採光率／２（式５）
したがって、前述の採光率の定義から、１個の画素ｌの遮光率は、以下の（式６）で示すように算出される。
画素ｌの遮光率＝１−画素ｌの採光率（式６） From (Equation 2), the light blocking rate of the pixel l2 is calculated as shown in (Equation 3) below.
The shading rate of the pixel l2 = l2 / N1 (Formula 3)
Here, if the value calculated by (1-light-shielding rate) is defined as the lighting rate, the lighting rate of the pixel 12 is
It is calculated as shown in (Equation 4) below.
The lighting rate of the pixel 12 = 1−the light blocking rate of the pixel 12 (Formula 4)
In contrast to the light blocking rate, which indicates the difficulty of entering the light beam, the lighting rate indicates the ease of entering the light beam. By the way, (Expression 4) is the daylighting rate in a state where two phase difference detection pixels are mixed. From this, the daylighting rate when the number of phase difference detection pixels is one is calculated as an average value of the daylighting rate in a state where two phase difference detection pixels are mixed. Therefore, the lighting rate of one pixel l is calculated as shown in the following (Formula 5).
Daylighting rate of pixel l = lighting rate of pixel l / 2/2 (Formula 5)
Therefore, from the above definition of the lighting rate, the shading rate of one pixel l is calculated as shown in the following (Equation 6).
Shading rate of pixel l = 1−lighting rate of pixel l (formula 6)

位相差検出画素の混入数が１個である画素については、ステップＳ１０５で計算された比率が遮光率として用いられる。一方、位相差検出画素の混入数が２個以上である画素の遮光率は、例えば混合に用いられた画素の遮光率の相加平均値である。ただし、撮像画素の採光率は１であるとする。この場合、例えば画素ｌ２の遮光率は、以下の（式１０）で示すようにして算出される。また、画素ｒ２ｌ１の遮光率は、以下の（式１１）で示すようにして算出される。
補正量換算処理の後、ゲイン補正部３１３のゲイン乗算処理部３１３２は、ゲイン補正処理を行う（ステップＳ１０９）。ゲイン補正処理においては、ステップＳ１０８の補正量換算処理で得られた遮光率から求められるゲイン補正量を位相差検出画素の画素出力に乗じる処理が行われる。これにより、位相差検出画素の画素出力が混入された画素における光量低下が補正される。したがって、画質劣化が補正される。 For pixels with one phase difference detection pixel, the ratio calculated in step S105 is used as the light shielding rate . On the other hand, the shading rate of a pixel having two or more phase difference detection pixels mixed therein is an arithmetic average value of the shading rates of the pixels used for mixing, for example. However, it is assumed that the lighting rate of the imaging pixel is 1. In this case, for example, the light blocking rate of the pixel 12 is calculated as shown in the following (Equation 10). Further, the shading rate of the pixel r211 is calculated as shown in the following (Equation 11).
After the correction amount conversion processing, the gain multiplication processing unit 3132 of the gain correction unit 313 performs gain correction processing (step S109). In the gain correction processing, processing for multiplying the pixel output of the phase difference detection pixel by the gain correction amount obtained from the light shielding rate obtained in the correction amount conversion processing in step S108 is performed. Thereby, the light amount decrease in the pixel mixed with the pixel output of the phase difference detection pixel is corrected. Therefore, image quality deterioration is corrected.

Claims

An image processing apparatus for processing a pixel output from an imaging device having an imaging pixel and a phase difference detection pixel,
A pixel mixing unit that mixes a plurality of the pixel outputs;
Output of the imaging pixel using one or more information of the number of mixed pixel outputs for each opening direction of the phase difference detection pixel for each mixed pixel output and the number of mixed pixel outputs for the imaging pixel And an image processing unit that corrects the mixed pixel output so that there is no deterioration in image quality due to the mixed pixel output of the phase difference detection pixel, and
An image processing apparatus comprising:

The said image processing part has a gain correction part which carries out gain correction of the pixel output which mixed the output of the said imaging pixel, and the output of the said phase difference detection pixel as correction | amendment of the said pixel output. Image processing apparatus.

The image processing unit uses at least one piece of information of the number of pixel outputs mixed for each opening direction of the phase difference detection pixels for each mixed pixel output and the number of pixel outputs mixed for the imaging pixels. The image processing apparatus according to claim 2, further comprising: a gain estimation unit that estimates a gain correction amount at the time of the gain correction in the gain correction unit so that image quality deterioration due to the phase difference detection pixel does not occur.

Information on the number of mixed pixel outputs for each opening direction of the phase difference detection pixels and information on the number of mixed pixel outputs of the imaging pixels are obtained by mixing the arrangement period of the phase difference detection pixels and the pixel output by the pixel mixing unit. The image processing apparatus according to claim 1, wherein the image processing apparatus is determined by a technique.

The image processing apparatus according to claim 4, wherein the pixel output mixing method is determined by a display form or a recording form of the pixel output.

When there is a pixel output including a pixel output of the phase difference detection pixel having a different opening direction in the mixed pixel output, the gain estimation unit outputs the pixel output of the phase difference detection pixel having a different opening direction. The image processing apparatus according to claim 3, wherein the gain correction amount is estimated without using an image.

The gain estimator may detect the phase difference with a different opening direction using simultaneous equations when the mixed pixel output includes a pixel output including a pixel output of the phase difference detection pixel with a different opening direction. The image processing apparatus according to claim 3, wherein the gain correction amount is estimated after obtaining a shading rate for each pixel.

The gain estimation unit may detect the phase difference with a different opening direction using four arithmetic operations when the mixed pixel output includes a pixel output including a pixel output of the phase difference detection pixel with a different opening direction. The image processing apparatus according to claim 3, wherein the gain correction amount is estimated after obtaining a shading rate for each pixel.

An image processing method for processing a pixel output from an imaging device having an imaging pixel and a phase difference detection pixel,
Mixing a plurality of the pixel outputs;
Output of the imaging pixel using one or more information of the number of mixed pixel outputs for each opening direction of the phase difference detection pixel for each mixed pixel output and the number of mixed pixel outputs for the imaging pixel And correcting the pixel output so that there is no image quality degradation due to the pixel output in which the outputs of the phase difference detection pixels are mixed ,
An image processing method.

The image processing method according to claim 9, wherein the correction of the pixel output is gain correction.

When correcting the pixel output, one or more pieces of information of the number of pixel outputs mixed for each aperture direction of the phase difference detection pixel for each of the mixed pixel outputs and the number of pixel outputs mixed for the imaging pixel The gain correction amount at the time of the gain correction is estimated so that there is no deterioration in image quality due to a pixel output in which the output of the imaging pixel and the output of the phase difference detection pixel are mixed using An image processing method described in 1.

An image processing program for processing a pixel output from an imaging device having an imaging pixel and a phase difference detection pixel,
A function of mixing a plurality of the pixel outputs;
Output of the imaging pixel using one or more information of the number of mixed pixel outputs for each opening direction of the phase difference detection pixel for each mixed pixel output and the number of mixed pixel outputs for the imaging pixel And a function of correcting the pixel output so that there is no deterioration in image quality due to the pixel output in which the outputs of the phase difference detection pixels are mixed , and
An image processing program for realizing a computer.

The image processing program according to claim 12, wherein the correction of the pixel output is gain correction.

When correcting the pixel output, one or more pieces of information of the number of pixel outputs mixed for each aperture direction of the phase difference detection pixel for each of the mixed pixel outputs and the number of pixel outputs mixed for the imaging pixel In order to cause the computer to further realize the function of estimating the gain correction amount at the time of gain correction so that there is no deterioration in image quality due to the pixel output in which the output of the imaging pixel and the output of the phase difference detection pixel are mixed. The image processing program according to claim 13.