JP6459183B2 - Imaging device - Google Patents

Description

  The present invention relates to an imaging apparatus.

  There is known a technique for interpolating deficient color component data at each pixel position with respect to image data captured through a color filter according to a Bayer array rule (see Patent Document 1).

Japanese Patent No. 5147777

  The prior art is a case where one photoelectric conversion unit (photodiode) is provided for each pixel, and has not been studied when a plurality of photoelectric conversion units are provided for each pixel.

The image sensor according to the present invention receives a first color component of subject light by a plurality of photoelectric conversion units in a pixel and outputs a plurality of first photoelectric conversion signals corresponding to the first color component. A plurality of second pixels that respectively receive a second color component of subject light by a plurality of photoelectric conversion units in the pixel and output a plurality of photoelectric conversion signals corresponding to the second color component; An image pickup device having an image sensor, a plurality of photoelectric conversion signals from the first pixel added for each first pixel, a plurality of photoelectric conversion signals from the second pixel added to each second pixel, and an addition by the adder An interpolation processing unit that interpolates a signal of a color component that is insufficient at each pixel position using a signal at a peripheral pixel position with respect to a target image composed of the first color component signal and the second color component signal If, before the addition by the addition section Among the photoelectric conversion signals of one of the plurality of photoelectric conversion units of the first pixel at the first position and the plurality of photoelectric conversion units of the first pixel at the second position different from the first position before addition by the addition unit The direction of the image structure is detected on the basis of the one photoelectric conversion signal, and the color component signal that is deficient at each pixel position of the target image is interpolated using the signals of the pixel positions around the detected direction. And a control unit for controlling the interpolation processing unit.

  According to the present invention, it is possible to appropriately interpolate when one pixel has a plurality of photoelectric conversion units.

1 is a block diagram illustrating a digital camera according to an embodiment of the present invention. It is a figure explaining the normal pixel provided in the image sensor. 3A is a diagram illustrating the arrangement of photoelectric conversion signals output from normal pixels, FIG. 3B is a diagram illustrating interpolation of G color component signals, and FIG. 3C is a diagram illustrating G after interpolation. It is a figure which illustrates the signal of a color component. 4A is a diagram obtained by extracting the R color component signal from FIG. 3A, FIG. 4B is a diagram illustrating interpolation of the color difference component Cr, and FIG. 4C is a diagram illustrating the signal of the color difference component Cr. It is a figure explaining interpolation. 5A is a diagram obtained by extracting the B color component signal from FIG. 3A, FIG. 5B is a diagram illustrating interpolation of the color difference component Cb, and FIG. 5C is a diagram illustrating the signal of the color difference component Cb. It is a figure explaining interpolation. It is a figure explaining the pixel for focus detection provided in the image sensor. It is a figure explaining the signal for an image based on the photoelectric conversion signal output from the pixel for focus detection. It is a figure explaining direction determination. It is a figure which shows the position of the attention pixel and surrounding pixel made into interpolation object.

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. FIG. 1 is a block diagram illustrating a digital camera 1 according to an embodiment of the present invention. The digital camera 1 is equipped with a photographing lens 2 as an imaging optical system. In the photographic lens 2, the focusing lens and the diaphragm are driven and controlled by the lens control unit 2 a that receives an instruction from the microprocessor 9. The photographing lens 2 forms a subject image on the imaging surface of the image sensor 3.

  The image pickup device 3 photoelectrically converts the subject image based on the drive signal from the image pickup control unit 4 that has received an instruction from the microprocessor 9. The photoelectric conversion signal output from the image sensor 3 is processed through the signal processing unit 5 and the A / D conversion unit 6 and then temporarily stored in the memory 7. Note that the image sensor 3 may include the signal processing unit 5 and the A / D conversion unit 6. Connected to the bus 8 are a lens control unit 2a, an imaging control unit 4, a memory 7, a microprocessor 9, a focus calculation unit (detection processing unit) 10, a recording unit 11, an image compression unit 12, an image processing unit 13, and the like. .

  An operation signal is input to the microprocessor 9 from an operation unit 9a including a release button. The microprocessor 9 sends an instruction to each block based on the operation signal from the operation unit 9a, and controls the photographing operation of the digital camera 1. The focus calculation unit 10 performs a known phase difference detection calculation using a photoelectric conversion signal output from the image sensor 3 and a focus detection signal, thereby adjusting the focus adjustment state (specifically, the imaging lens 2). Defocus amount) is detected. The focus detection signal is a photoelectric conversion signal output from a focus detection pixel 32 (FIG. 4) described later. The microprocessor 9 instructs the lens control unit 2a to drive the focusing lens according to the defocus amount.

  The image processing unit 13 performs predetermined image processing including interpolation processing, which will be described later, on the photoelectric conversion signal output from the image sensor 3 and the image signal. In the present embodiment, a photoelectric conversion signal output from a photoelectric conversion unit of a normal pixel 31 (FIG. 2), which will be described later, and a photoelectric conversion signal output from a focus detection pixel 32 (FIG. 4) within the same pixel. A photoelectric conversion signal obtained by adding signals output from a plurality of photoelectric conversion units is used as an image signal.

  The image compression unit 12 compresses the image data after image processing in a predetermined format. The recording unit 11 records the compressed image data on the recording medium 11a in a predetermined file format. The recording medium 11 a is configured by a memory card that is detachable from the recording unit 11.

<Normal pixel>
FIG. 2 is a diagram illustrating the normal pixel 31 provided in the image sensor 3. In FIG. 2, normal pixels 31 are arranged in a two-dimensional square lattice pattern in the row direction (horizontal) and the column direction (vertical). One normal pixel 31 is configured by one photoelectric conversion unit (photodiode: PD) provided under one microlens (not shown). Color filters (R: red filter, G: green filter, B: blue filter) are arranged at each pixel position according to the rules of the Bayer array.

  Since the signal of the color component different from the color component of the arranged color filter is insufficient at each pixel position, the image processing unit 13 generates a signal of the insufficient color component using the signal at the peripheral pixel position. Perform array interpolation (demosaic). Specifically, in the case of the position of the G color component pixel, there is no R color component signal and no B color component signal, so that the R color component signal at the pixel position around the G color component pixel is used for R. A color component signal is generated, and a B color component signal is generated using a B color component signal at pixel positions around the G color component pixel.

  Similarly, since there is no G color component signal and B color component signal in the case of the R color component pixel position, the G color component signal is used by using the G color component signal at the pixel positions around the R color component pixel. The B color component signal is generated using the B color component signal at the pixel positions around the R color component pixel.

  Further, in the case of the pixel position of the B color component, since the signal of the G color component and the signal of the R color component do not exist, the G color component signal at the pixel position around the B color component pixel is used. A signal is generated, and an R color component signal is generated using an R color component signal at a pixel position around the B color component pixel. The interpolation processing in the case of such a normal pixel 31 is known, but an example will be described below.

<G color interpolation>
FIG. 3A is a diagram illustrating the arrangement of photoelectric conversion signals output from the normal pixels 31. Corresponding to each pixel position, it has one of R, G, and B color components according to the rules of the Bayer array. The image processing unit 13 that performs G color interpolation sets the positions of the R color component and the B color component as the target position in order, and uses the photoelectric conversion signals of the four G color components positioned around the target position to perform G at the target position. A color component signal is generated by interpolation processing. For example, when a G color component signal is interpolated at a target position indicated by a thick frame in FIG. 3B, four G color component photoelectric conversion signals located in the vicinity of the target position are used. The image processing unit 13 uses, for example, (aG1 + bG2 + cG3 + dG4) / 4 as a G color component signal at the target position. Note that a to d are weighting coefficients provided according to the distance between the position in the vicinity and the target position and the image structure.

  The image processing unit 13 performs a process of interpolating the G color component signal at each of the B color component position and the R color component position, so that the image processing unit 13 performs the process at each position of the normal pixel 31 as shown in FIG. A signal of G color component can be obtained.

<R color interpolation>
FIG. 4A is a diagram obtained by extracting R color component signals from FIG. The image processing unit 13 that performs R color interpolation uses the positions of the B color component and the G color component in FIG. 3A as the target position in order, and outputs photoelectric conversion signals of four R color components positioned around the target position. The R color component signal at the target position is generated by interpolation processing. The image processing unit 13 calculates a color difference component Cr signal shown in FIG. 4B based on the G color component signal shown in FIG. 3C and the R color component signal shown in FIG. .

  Further, when the color difference component Cr signal is interpolated at the target position indicated by the thick frame in FIG. 4B, the image processing unit 13 uses four color difference component Cr signals located in the vicinity of the target position. For example, the image processing unit 13 sets (eCr1 + fCr2 + gCr3 + hCr4) / 4 as a signal of the color difference component Cr at the target position. Note that e to h are weighting coefficients provided according to the distance between the position in the vicinity and the target position and the image structure.

  Similarly, when the color difference component Cr signal is interpolated at the target position indicated by the thick frame in FIG. 4C, the image processing unit 13 uses four color difference component Cr signals located in the vicinity of the target position. For example, the image processing unit 13 sets (qCr2 + rCr4 + sCr5 + tCr6) / 4 as a signal of the color difference component Cr at the target position. Note that q to t are weighting coefficients provided according to the distance between the position in the vicinity and the position of interest and the image structure.

  The image processing unit 13 obtains the color difference component Cr signal at each position of the normal pixel 31, and then adds the G color component signal shown in FIG. An R color component signal can be obtained at each of the 31 positions.

<B color interpolation>
FIG. 5A is a diagram obtained by extracting the B color component signal from FIG. The image processing unit 13 that performs B color interpolation uses the positions of the R color component and the G color component in FIG. 3A as the target position, and uses the photoelectric conversion signals of the four B color components positioned around the target position. A B color component signal at the position of interest is generated by interpolation processing. The image processing unit 13 calculates a color difference component Cb signal shown in FIG. 5B based on the G color component signal shown in FIG. 3C and the B color component signal shown in FIG. .

  Further, when the signal of the color difference component Cb is interpolated at the target position indicated by the thick frame in FIG. 5B, the image processing unit 13 uses the signals of the four color difference components Cb located in the vicinity of the target position. For example, the image processing unit 13 sets (uCb1 + vCb2 + wCb3 + xCb4) / 4 as a signal of the color difference component Cb at the target position. Note that u to x are weighting coefficients provided according to the distance between the position in the vicinity and the target position and the image structure.

  Similarly, when the signal of the color difference component Cb is interpolated at the target position indicated by the thick frame in FIG. 5C, the image processing unit 13 uses the signals of the four color difference components Cb located in the vicinity of the target position. For example, the image processing unit 13 uses (yCb2 + zCb4 + αCb5 + βCb6) / 4 as a signal of the color difference component Cb at the target position. Note that y, z, α, and β are weighting coefficients provided in accordance with the distance between the nearby position and the target position and the image structure.

  The image processing unit 13 obtains the color difference component Cb signal at each position of the normal pixel 31, and then adds the G color component signal shown in FIG. B color component signals can be obtained at the 31 positions.

<Focus detection pixel>
FIG. 6 is a diagram illustrating the focus detection pixels 32 provided in the image sensor 3. A focus detection pixel 32 is disposed instead of the normal pixel 31 at a position corresponding to the AF area used by the focus calculation unit 10 to detect the focus adjustment state on the image pickup surface of the image pickup device 3. In FIG. 6, the focus detection pixel 32 includes two photoelectric conversion units (photodiodes: PD) provided under one microlens (not shown). In the case of a 2PD configuration having two photoelectric conversion units per pixel, two photoelectric conversion units are arranged in a horizontal direction (referred to as horizontal division), and two photoelectric conversion units are arranged in a vertical direction (referred to as vertical division). ) And exist. FIG. 6 is a diagram illustrating the case of horizontal division. In the case of vertical division, the row direction and the column direction may be interchanged.

  Light that has entered through the different regions of the photographic lens 2 is incident on the two photoelectric conversion units in the focus detection pixel 32. That is, a pair of subject light beams used by the focus calculation unit 10 for phase difference detection calculation are incident.

  As described above, in the present embodiment, the photoelectric conversion signal output from the photoelectric conversion unit of the normal pixel 31 and the photoelectric conversion signal output from the focus detection pixel 32 and output from a plurality of photoelectric conversion units in the same pixel. The photoelectric conversion signal obtained by adding the obtained signals is handled as an image signal. For this reason, when light of the same intensity is incident on the focus detection pixel 32 and the normal pixel 31, the sum of the photoelectric conversion signals output from the two photoelectric conversion units in the focus detection pixel 32 is the normal pixel 31. The size of the two photoelectric conversion units provided in the focus detection pixel 32 and the size of one photoelectric conversion unit provided in the normal pixel 31 are optimized so that the level is equivalent to the photoelectric conversion signal output from ing.

  FIG. 7 is a diagram illustrating an image signal based on the photoelectric conversion signals output from the two photoelectric conversion units of the focus detection pixel 32. Since the two photoelectric conversion units at one pixel position receive light through the same color filter, the two photoelectric conversion signals are added to generate an image signal representing the pixel. Similarly to the normal pixel 31, color filters are arranged also on the focus detection pixel 32 according to the Bayer array rules. For this reason, the image processing unit 13 performs color filter array interpolation on the photoelectric conversion signal after addition as shown in FIG. 7 to generate a signal of a color component that is insufficient at each pixel position by using a signal at a surrounding pixel position. .

In other words, the procedure of the interpolation process is
1. 1. Direction determination based on photoelectric conversion signal before addition Interpolation is performed in the order based on the photoelectric conversion signal after the addition.

<Direction determination>
FIG. 8 is a diagram for explaining the direction determination, and is an enlarged view of a part of the array of photoelectric conversion signals output from the focus detection pixels 32 of FIG. In FIG. 8, positions (i, j) and (i + 1, j) corresponding to two photoelectric conversion units (hereinafter referred to as attention positions) at pixel positions to be interpolated are indicated by diagonal lines and horizontal lines, respectively. The image processing unit 13 determines the direction of the image structure with respect to an insufficient color component (eg, G component) at each of the positions of interest (i, j) and (i + 1, j). The image processing unit 13 includes six positions (i−1, j), (i + 2, j), (i, j−1) around the positions of interest (i, j) and (i + 1, j). , (I, j + 1), (i + 1, j-1), and (i + 1, j + 1) are used to determine the direction by the following equations (1) to (8). . In FIG. 8, the above six positions are indicated by circles.

ただし、閾値th_1およびth_2 は所定値であり、ノイズによる光電変換信号の揺らぎ幅の数倍程度に設定する。

However, the thresholds th_1 and th_2 are predetermined values, and are set to about several times the fluctuation width of the photoelectric conversion signal due to noise.

  When the above equation (1) is established, the image processing unit 13 determines that the image structure is the horizontal (row) direction with respect to the G color component at the target position (i, j).

  When the above equation (2) is established, the image processing unit 13 determines that the image structure is the horizontal (row) direction with respect to the G color component at the target position (i + 1, j).

  When the above equation (3) is established, the image processing unit 13 determines that the image structure is the vertical (column) direction with respect to the G color component at the target position (i, j).

  When the above equation (4) is established, the image processing unit 13 determines that the image structure is the vertical (column) direction with respect to the G color component at the target position (i + 1, j).

  When the above equation (5) is established, the image processing unit 13 determines that there is a corner of the image structure (there is an edge corner) for the G color component at the target position (i, j).

  When the above equation (6) is established, the image processing unit 13 determines that there is no image structure (not on the edge) for the G color component at the target position (i, j).

  When the above equation (7) is established, the image processing unit 13 determines that there is a corner of the image structure (there is an edge corner) for the G color component at the target position (i + 1, j).

  When the above equation (8) is established, the image processing unit 13 determines that there is no image structure (not on the edge) regarding the G color component at the target position (i + 1, j).

<When determination results are the same at two positions of interest>
When the determination result at the target position (i, j) and the determination result at the target position (i + 1, j) are the same, the image processing unit 13 uses the determination result as the image structure at the pixel position to be interpolated. Adopt as information to show.

<When the determination results are different at the two positions of interest>
(A) The image processing unit 13 employs a method that strongly indicates directionality when the determination result at the target position (i, j) is different from the determination result at the target position (i + 1, j). Specifically, when it is determined that one of the attention positions is in the horizontal (row) direction or the vertical (column) direction, and the other attention position is determined to have a corner of the image structure or no image structure, the horizontal (row) The determination result determined as the (row) direction or the vertical (column) direction is employed as information indicating the image structure at the pixel position to be interpolated.

(B) When the image processing unit 13 determines the horizontal (row) direction at one target position and determines the vertical (column) direction at the other target position, the image processing unit 13 employs a method that strongly indicates directionality. Specifically, a determination result having a larger absolute value of the difference in the above formulas (1) to (4) is adopted as information indicating the image structure at the pixel position to be interpolated. For example, if the absolute value of the difference in the previous stage of the expression is greater than the absolute value of the difference in the subsequent stage, it is determined as the vertical (column) direction, and the absolute value of the difference in the subsequent stage of the expression is greater than the absolute value of the difference in the previous stage. In this case, the horizontal (row) direction is determined.

<Addition>
As shown in FIG. 9, the image processing unit 13 adds the photoelectric conversion signals from the two photoelectric conversion units at each pixel position of the focus detection pixel 32. FIG. 9 is a diagram illustrating the pixel of interest (I, J) to be interpolated and the positions of the peripheral pixels of the pixel of interest (I, J), and corresponds to an enlarged view of a part of FIG. In FIG. 9, the pixel corresponding to the positions of interest (i, j) and (i + 1, j) in FIG. 8 is designated as the pixel of interest (I, J). That is, the photoelectric conversion signal after addition at the target pixel (I, J) is R (I, J) = R (i, j) + R (i + 1, j).
When the image pickup device 3 has an addition function, the photoelectric conversion signal after the addition may be output from the image pickup device 3.

<G color interpolation>
In FIG. 9, the image processing unit 13 interpolates insufficient color components for the pixel of interest (I, J). In the interpolation of the G component, instead of the calculation formula for the normal pixel 31 described with reference to FIG. 3B, the pixel of interest (I, J) and the four pixel positions around the pixel of interest (I-1 , J), (I, J-1), (I, J + 1), and (I + 1, J) photoelectric conversion signals are used according to the following equation (9). In FIG. 9, the four positions are indicated by circles.

ただし、mは縦（列）方向の重み係数であり、nは横（行）方向の重み係数である。

Here, m is a weighting factor in the vertical (column) direction, and n is a weighting factor in the horizontal (row) direction.

  When the image processing unit 13 determines the horizontal (row) direction in the direction determination, the image processing unit 13 decreases the weighting coefficient m in Expression (9) and increases the weighting coefficient n.

  When the image processing unit 13 determines the vertical (column) direction in the direction determination, the image processing unit 13 decreases the weighting factor n in Expression (9) and increases the weighting factor m.

  When the image processing unit 13 determines that there is a corner of the image structure or no image structure in the direction determination, the image processing unit 13 sets the weighting coefficient m = n in Expression (9), respectively. Through the above interpolation processing, a G color component signal can be obtained at each position of the focus detection pixel 32.

<R color and B color interpolation>
The R color interpolation and the B color interpolation are performed in the same manner as in the case of the normal pixel 31 using the signal of the G color component at each position of the focus detection pixel 32 described above. When determining the weighting factor, the result of the direction determination can be used. For example, when the determination direction of the image structure is the horizontal (row) direction, the weighting factors for the signals on the left and right of the target position in FIGS. 4 and 5 are increased, and the weighting factors for the signals above and below the target position are decreased. When the image structure determination direction is the vertical (column) direction, the weighting factors for the signals above and below the target position in FIGS. 4 and 5 are increased, and the weighting factors for the signals at the left and right of the target position are decreased.

According to the embodiment described above, the following operational effects can be obtained.
(1) The digital camera 1 receives the first color component (G) of the subject light by a plurality of photoelectric conversion units in the pixel, and outputs a plurality of photoelectric conversion signals corresponding to the first color component, respectively. The plurality of first pixels and the second color component (B, R) of the subject light are respectively received by the plurality of photoelectric conversion units in the pixel, and a plurality of photoelectric conversion signals corresponding to the second color component are respectively output. The image sensor 3 having a plurality of second pixels, and a plurality of photoelectric conversion signals by the first pixel are added for each first pixel, and a plurality of photoelectric conversion signals by the second pixel are added for each second pixel And a color component signal that is insufficient at each pixel position with respect to a target image composed of the first color component signal and the second color component signal after addition by the image processing unit 13. Interpolate using signals at surrounding pixel locations Based on the image processing unit 13 and a plurality of photoelectric conversion signals before addition by the image processing unit 13, the direction of the image structure is detected, and the color component signal that is insufficient at each pixel position of the target image is detected in the vicinity of the direction of the detected direction. And a microprocessor 9 that controls the image processing unit 13 so as to interpolate using the signal of the pixel position. By having a plurality of photoelectric conversion units in the pixel, the direction of the image structure can be determined more finely than in the case where light is received by one photoelectric conversion unit in the pixel, so that appropriate interpolation is performed while maintaining the image structure. be able to.

(2) Since the image processing unit 13 interpolates the signal after the addition in the interpolation target pixel using the signal of the pixel position along the detected direction among the pixel positions around the interpolation target pixel, the direction determined with high precision Can be appropriately interpolated using the signals of the surrounding pixel positions along the line.

(3) Each of the first pixels receives the light of the first color component by the two photoelectric conversion units and outputs two photoelectric conversion signals, and each of the second pixels receives the light of the second color component. Since the two photoelectric conversion units respectively receive light and output two photoelectric conversion signals, it is possible to perform appropriate interpolation while maintaining the image structure in the so-called 2PD structure.

(4) When there are two directions detected at the target pixel position, the microprocessor 9 controls the image processing unit 13 so as to adopt the direction detected strongly among the two, so that it is appropriate according to the image structure. Interpolation can be performed.

(5) The first pixel and the second pixel are focus detection pixels 32, and the plurality of photoelectric conversion units in the pixels receive light beams that have passed through different areas of the photographic lens 2 and receive a plurality of photoelectric conversion signals. Output it. Thereby, when the photoelectric conversion signal in the pixel in the focus detection pixel 32 is added to obtain the image signal, it is possible to perform appropriate interpolation while maintaining the image structure.

The following modifications are also within the scope of the present invention, and one or a plurality of modifications can be combined with the above-described embodiment.
(Modification 1)
In the above-described embodiment, an example has been described in which the direction that strongly indicates directionality is employed as a representative when the determination result is different at two attention positions in the direction determination (above (B)). A direction in which two directions are combined may be adopted. As a result of the synthesis, in the case of 45 degrees oblique to the horizontal (row) direction or 135 degrees oblique to the horizontal (row) direction, the image processing unit 13 reflects in the interpolation calculation as follows.

  For example, when determining the weight coefficient in the R color interpolation and the B color interpolation described with reference to FIGS. 4 and 5, the determination result of 45 degrees or 135 degrees can be used. When the image structure determination direction is 45 degrees diagonally, the image processing unit 13 increases the weighting coefficients for the signals at the lower left and upper right of the target position in FIGS. 4B and 5B, and sets the upper left of the target position. Decrease the weighting factor for the lower right signal. Further, when the image structure determination direction is oblique 135 degrees, the weighting factors for the signals at the upper left and lower right of the target position in FIGS. 4B and 5B are increased, and the signals at the lower left and upper right of the target position are displayed. Reduce the weighting factor for.

  According to the first modification, the interpolation process can be performed by utilizing the determined two directions.

(Modification 2)
In the above embodiment, the direction of the image structure is determined based on the photoelectric conversion signals before the addition by the two photoelectric conversion units of the focus detection pixels 32, and the photoelectric conversion signals after the addition are determined according to the determined directions. An example of performing interpolation processing has been described. On the other hand, interpolation processing may be performed on the photoelectric conversion signal before addition in accordance with the determined direction of the image structure.

<G color interpolation>
The image processing unit 13 according to Modification 2 interpolates the insufficient color components at each of the target positions (i, j) and (i + 1, j) in FIG. The G component interpolation is performed at six positions (i-1, j), (i + 2, j), (i, j-1) around the positions of interest (i, j) and (i + 1, j). , (I, j + 1), (i + 1, j-1), and (i + 1, j + 1) photoelectric conversion signals are used according to the following equations (10) and (11).

ただし、mおよびoは縦（列）方向の重み係数であり、nおよびｐは横（行）方向の重み係数である。

However, m and o are weighting factors in the vertical (column) direction, and n and p are weighting factors in the horizontal (row) direction.

  When the above equation (1) is established, the image processing unit 13 determines that the image structure is in the horizontal (row) direction with respect to the G color component at the position of interest (i, j), and sets the weighting coefficient m in the equation (10). Decrease and increase weighting factor n.

  When the above equation (2) is established, the image processing unit 13 determines that the image structure is in the horizontal (row) direction with respect to the G color component at the target position (i + 1, j), and the weighting coefficient of the equation (11) Decrease o and increase weighting factor p.

  When the above equation (3) is established, the image processing unit 13 determines that the image structure is the vertical (column) direction with respect to the G color component at the target position (i, j), and sets the weighting coefficient m of the equation (10) to Increase to decrease weighting factor n.

  When the above equation (4) is established, the image processing unit 13 determines that the image structure is the vertical (column) direction with respect to the G color component at the target position (i + 1, j), and the weighting coefficient of the equation (11) Increase o to decrease weighting factor p.

  When the above equation (5) holds, the image processing unit 13 determines that there is a corner of the image structure (there is an edge corner) with respect to the G color component at the target position (i, j), and the equation (10) The weighting coefficient m = n.

  When the above equation (6) is established, the image processing unit 13 determines that there is no image structure (not on the edge) for the G color component at the target position (i, j), and the weighting coefficient of the equation (10) Let m = n.

  When the above expression (7) is established, the image processing unit 13 determines that there is a corner of the image structure (there is an edge corner) with respect to the G color component at the target position (i + 1, j), and the expression ( 10) The weighting factor o = p.

  When the above equation (8) holds, the image processing unit 13 determines that there is no image structure (not on the edge) for the G color component at the target position (i + 1, j), and the equation (11) The weighting factor o = p. Through the above interpolation processing, a G color component signal can be obtained at each position of the photoelectric conversion signal before addition by the two photoelectric conversion units of the focus detection pixel 32.

<R color and B color interpolation>
The R color interpolation and the B color interpolation are performed using the G color component signal described above. When determining the weighting factor, the result of the direction determination can be used. For example, when the determination direction of the image structure is the horizontal (row) direction, the weighting factors for the signals on the left and right of the target position in FIG. When the image structure determination direction is the vertical (column) direction, the weighting factors for the signals above and below the target position in FIG. 8 are increased, and the weighting factors for the left and right signals at the target position are decreased.

  Through the above interpolation processing, R color and B color component signals can be obtained at each position of the photoelectric conversion signal before addition by the two photoelectric conversion units of the focus detection pixel 32. According to Modification 2, it is possible to obtain interpolated images corresponding to the positions of the two photoelectric conversion units of the focus detection pixel 32, respectively.

(Modification 3)
As an example of the focus detection pixel 32, a 2PD configuration having two photoelectric conversion units per pixel has been described as an example. However, the number of the plurality of photoelectric conversion units is not limited to two, and may be 4PD or 16PD.

(Modification 4)
As an example of the image sensor 3, the example in which the focus detection pixels 32 are arranged only at positions corresponding to the AF areas has been described. However, the focus detection pixels 32 may be arranged over the entire imaging surface. Good.

  Although various embodiments and modifications have been described above, the present invention is not limited to these contents. Other embodiments conceivable within the scope of the technical idea of the present invention are also included in the scope of the present invention.

DESCRIPTION OF SYMBOLS 1 ... Digital camera 2 ... Shooting lens 3 ... Imaging device 9 ... Microprocessor 13 ... Image processing part 31 ... Normal pixel 32 ... Pixel for focus detection

Claims (8)

A plurality of first pixels each receiving a first color component of subject light by a plurality of photoelectric conversion units in the pixel and outputting a plurality of photoelectric conversion signals corresponding to the first color component; and the subject light A plurality of second pixels that respectively receive a second color component of the first color component by a plurality of photoelectric conversion units in the pixel and output a plurality of photoelectric conversion signals corresponding to the second color component; ,
An adder that adds the plurality of photoelectric conversion signals by the first pixel for each of the first pixels and adds the plurality of photoelectric conversion signals by the second pixel for each of the second pixels;
For the target image composed of the signal of the first color component and the signal of the second color component after the addition by the adding unit, the signal of the color component that is insufficient at each pixel position is signaled at the surrounding pixel positions. An interpolation processing unit that performs interpolation using
One photoelectric conversion signal of the plurality of photoelectric conversion units of the first pixel at the first position before addition by the addition unit, and a second position different from the first position before addition by the addition unit The direction of the image structure is detected based on one photoelectric conversion signal among the plurality of photoelectric conversion units of the first pixel, and the direction of the color component signal that is insufficient at each pixel position of the target image is detected. A control unit that controls the interpolation processing unit to perform interpolation using signals of the peripheral pixel positions of
That includes a imaging device. The imaging device according to claim 1,
The control unit includes one photoelectric conversion signal among the plurality of photoelectric conversion units of the first pixel at the first position before addition by the addition unit, and the first position before addition by the addition unit. An imaging apparatus that detects a direction of an image structure related to a color component different from the first color component based on one photoelectric conversion signal among a plurality of photoelectric conversion units of the first pixel at different second positions. The imaging device according to claim 2,
The control unit includes one photoelectric conversion signal of the plurality of photoelectric conversion units of the second pixel at the third position before addition by the addition unit, and the third position before addition by the addition unit. An imaging device that detects a direction of an image structure related to a color component different from the second color component based on one photoelectric conversion signal of the plurality of photoelectric conversion units of the second pixel at a different fourth position. In the imaging device according to any one of claims 1 to 3 ,
The interpolation processing section, the interpolation of pixel positions around the target pixel using the signal of the pixel position along the detected direction, wherein the interpolation target you interpolate the signal after the addition of the pixel imaging device. In the imaging device according to any one of claims 1 to 4 ,
Each of the first pixels receives light of the first color component by two photoelectric conversion units, and outputs the two photoelectric conversion signals.
The second pixel, respectively, the second light to the two photoelectric conversion portions each received to two of the photoelectric conversion signal you output imaging device of a color component. The imaging apparatus according to claim 5 ,
Wherein, when a direction where the detected at a pixel location of interest is two, that controls the interpolation processing unit to adopt a direction detected strongly among the two imaging device. The imaging apparatus according to claim 5 ,
Wherein, when a direction where the detected at a pixel location of interest is two, that controls the interpolation processing unit to adopt a direction combining the two directions imaging device. In the imaging device according to any one of claims 1 to 7 ,
The first pixel and the second pixel are focus detection pixels,
The plurality of photoelectric conversion units of the first pixel and the second in the pixel, a plurality of photoelectric conversion signal to that imaging device outputs by receiving the light beams passing through different regions of each imaging lens.

Images