JP2019154039A - Imaging device and imaging apparatus - Google Patents

Abstract

To provide an interpolation method with improved resolution in ranging pixels.SOLUTION: The imaging device includes: a plurality of pixels arranged two-dimensionally; and a signal processing unit for calculating each pixel value in an image by applying smoothing and sharpening filters to outputs of a plurality of pixels. The plurality of pixels includes: a plurality of first pixels, provided continuously in a first direction, for photoelectrically converting incident light of a first wavelength; and a plurality of second pixels, provided continuously in the first direction with a half-pitch shift with respect to the plurality of first pixels, for photoelectrically converting incident light having a second wavelength.SELECTED DRAWING: Figure 4

Description

  The present invention relates to an imaging element and an imaging apparatus.

Conventionally, a technique for converting an output signal of an image sensor having a non-Bayer array into a signal of a Bayer array is known. (For example, refer to Patent Document 1).
[Prior art documents]
[Patent Literature]
[Patent Document 1] JP2013-254076A

  However, since the output signals of two adjacent pixels are added to generate one pixel signal in the Bayer array, the resolution decreases.

  In the first aspect of the present invention, each pixel value in an image is calculated by applying a smoothing filter and a sharpening filter to a plurality of pixels arranged in two dimensions and the output of the plurality of pixels. A plurality of pixels provided continuously in the first direction, and a plurality of first pixels that photoelectrically convert incident light having the first wavelength, and a plurality of pixels that are continuous in the first direction. There is provided an imaging device having a plurality of second pixels that are photoelectrically converted from incident light having a second wavelength, which is provided with a half-pitch shift relative to the first pixel.

  According to a second aspect of the present invention, there is provided an imaging device comprising the above-described imaging device.

  It should be noted that the above summary of the invention does not enumerate all the necessary features of the present invention. In addition, a sub-combination of these feature groups can also be an invention.

2 is a cross-sectional view of a single-lens reflex camera 400. FIG. 2 is a block diagram illustrating a configuration example of an image sensor 200. FIG. It is the figure which expanded the one part area | region of the imaging chip 10 in 1st Embodiment. FIG. 6 is a diagram showing a case where there is continuity in a first direction in a determination / interpolation region 40; FIG. 10 is a diagram showing a case where there is continuity in the second direction in the determination / interpolation region 40; FIG. 6 is a diagram showing a determination / interpolation area 41. It is a figure which shows the correspondence of each pixel in the determination and interpolation area | region 40, and the position of the pixel value produced | generated by interpolation. 4 is a schematic diagram showing pixel values interpolated and generated at a position corresponding to a determination / interpolation region 40. FIG. It is a flowchart 70 explaining the process which switches the addition mode and filter processing mode in 2nd Embodiment.

  Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, not all the combinations of features described in the embodiments are essential for the solving means of the invention.

  FIG. 1 is a cross-sectional view of a single-lens reflex camera 400. A single-lens reflex camera 400 as an example of an imaging apparatus includes an imaging device 200, a lens unit 500, and a camera body 600. A lens unit 500 is attached to the camera body 600. The lens unit 500 includes an optical system arranged along the optical axis 410 in the lens barrel, and guides an incident subject light flux to the image sensor 200 of the camera body 600.

  In this example, a single-lens reflex camera 400 will be described as an example of an imaging apparatus, but the camera body 600 may be regarded as an imaging apparatus. Further, the imaging device is not limited to an interchangeable lens camera provided with a mirror unit, but may be an interchangeable lens camera without a mirror unit, or a lens integrated camera regardless of the presence or absence of the mirror unit. In this example, the direction in which the subject luminous flux travels along the optical axis 410 is defined as the third direction. The directions perpendicular to the third direction and perpendicular to each other are defined as a first direction and a second direction.

  A lens mount 550 is coupled to the camera body 600. The camera body 600 includes a mirror 672 on the third direction side of the body mount 660. The mirror 672 is fixed to the shaft so as to be rotatable between an oblique position provided to be inclined with respect to the subject light beam incident in the third direction from the lens unit 500 and a retracted position for retreating from the subject light beam. .

  When the mirror 672 is in the oblique position, most of the subject light beam incident through the lens unit 500 is reflected by the mirror 672 and guided to the focus plate 652. The focus plate 652 is arranged at a position conjugate with the light receiving surface of the image sensor 200 and visualizes the subject image formed by the optical system of the lens unit 500. The subject image formed on the focus plate 652 is observed from the viewfinder 650 through the pentaprism 654 and the viewfinder optical system 656.

  The focus plate 652, the pentaprism 654, and the mirror 672 are supported by a mirror box 670 as a structure. The image sensor 200 is attached to the mirror box 670. When the mirror 672 is retracted to the retracted position and the front curtain and rear curtain of the shutter unit 340 are in the open state, the subject luminous flux that passes through the lens unit 500 reaches the light receiving surface of the image sensor 200.

  On the side opposite to the shutter unit 340 with respect to the image sensor 200, a body substrate 620 and a rear display unit 634 are sequentially arranged. A rear display unit 634 employing a liquid crystal panel or the like is located on the rear surface of the camera body 600. Electronic circuits such as a CPU 622 and an image processing unit 624 are mounted on the body substrate 620. The output of the image sensor 200 is delivered to the image processing unit 624.

  FIG. 2 is a block diagram illustrating a configuration example of the image sensor 200. The imaging element 200 includes an imaging chip 10 and a signal processing chip 30. A light beam enters the imaging chip 10 from the third direction. The imaging chip 10 is stacked on the signal processing chip 30 in the third direction. The imaging chip 10 is electrically connected to the signal processing chip 30 through the bumps 15. The imaging chip 10 has a plurality of photoelectric conversion units arranged two-dimensionally in the first direction and the second direction. Each of the plurality of photoelectric conversion units photoelectrically converts incident light. That is, each of the plurality of photoelectric conversion units generates an electric charge according to the amount of light flux incident from the third direction. Note that the photoelectric conversion unit functions as a part of the pixel. The pixel may process the output of the photoelectric conversion unit. The imaging chip 10 outputs the converted analog image signal to the signal processing unit 32 of the signal processing chip 30.

  The signal processing chip 30 includes a signal processing unit 32, an AD conversion unit 34, and a focus detection unit 36. The signal processing unit 32 performs predetermined analog signal processing on the analog image signal output from the imaging chip 10. The signal processing unit 32 includes an amplifier circuit and a CDS (correlated double sampling) circuit. The amplifier circuit amplifies the signal voltage value of the analog image signal. In addition, the CDS circuit corrects variations in the signal voltage value due to noise.

  The AD converter 34 converts the analog image signal output from the signal processor 32 into a digital image signal. The AD conversion unit 34 outputs the digital image signal to the signal processing unit 32 and the focus detection unit 36.

  The signal processing unit 32 performs a specific calculation on the digital image signal output from the AD conversion unit 34. The signal processing unit 32 has a filter processing mode and an addition processing mode. In the filter processing mode, the signal processing unit 32 applies directionality determination, a smoothing filter, and a sharpening filter to a digital image signal. The signal processing unit 32 applies the filter to calculate each pixel value in the image. Further, in the addition processing mode, the signal processing unit 32 adds digital image signals to calculate each pixel value in the image. The signal processing unit 32 may select one of the filter processing mode and the addition processing mode. Information on the pixel value calculated by the signal processing unit 32 is output to the image processing unit 624. That is, the signal processing unit 32 preprocesses the image before outputting the image to the image processing unit 624.

  The focus detection unit 36 detects the focus position of the optical system through which the incident light has passed. Specifically, the focus detection unit 36 detects the image shift amount of the pair of images based on the digital image signal. Further, the focus detection unit 36 performs a calculation according to the image shift amount, and calculates a defocus amount that is a deviation of the current image plane from the planned image plane. In order to adjust the defocus amount, the position of the lens unit 500 of the single-lens reflex camera 400 is adjusted.

  The image processing unit 624 includes an image processing ASIC 625 and a recording unit 60. The image processing ASIC 625 performs various image processing using the recording unit 60 as a work space, and generates image data. When generating image data in the JPEG file format, the image processing ASIC 625 executes compression processing after performing white balance processing, gamma processing, and the like. The generated image data is recorded in the recording unit 60. The rear display unit 634 displays an image corresponding to the image data recorded in the recording unit 60 for a preset time.

  FIG. 3 is an enlarged view of a partial region of the imaging chip 10 according to the first embodiment. In this example, each of the plurality of photoelectric conversion units 17 is a square having diagonal lines parallel to the first and second directions. Each photoelectric conversion unit 17 functions as a part of each pixel. One photoelectric conversion unit 17 that functions as a part of one pixel is arranged in a matrix in the first direction and the second direction. Note that one pixel may also have a signal processing function for processing the output of the photoelectric conversion unit 17. The photoelectric conversion unit 17 is provided with the microlens 12 so as to overlap in the third direction. In FIG. 3, the microlens 12 is indicated by a circle. One microlens 12 guides light incident through the lens unit 500 to the photoelectric conversion unit 17 provided correspondingly. The photoelectric conversion unit 17 of this example may be a photodiode provided at a position overlapping the microlens 12.

  The plurality of photoelectric conversion units 17 are provided with filters that transmit red (R), green (G), and blue (B) wavelengths, respectively. The photoelectric conversion unit 17-G provided with a filter that transmits incident light having a green (G) wavelength is continuously provided in the first direction and the second direction. The photoelectric conversion unit 17-R provided with a filter that transmits incident light having a red (R) wavelength is continuously provided in the first direction. A photoelectric conversion unit 17-B provided with a filter that transmits incident light having a blue (B) wavelength is also provided continuously in the first direction. The plurality of photoelectric conversion units 17-R and 17-B are provided with a half-pitch shift with respect to the plurality of photoelectric conversion units 17-G. In this example, the first photoelectric conversion unit is the photoelectric conversion unit 17-G, the second photoelectric conversion unit is the photoelectric conversion unit 17-B, and the third photoelectric conversion unit is the photoelectric conversion unit 17-R. Will be described. However, in another example, the second photoelectric conversion unit may be the photoelectric conversion unit 17-R, and the third photoelectric conversion unit may be the photoelectric conversion unit 17-B.

  In this example, the half pitch means half the length of the diagonal line parallel to the first direction in the photoelectric conversion unit 17 and half the length of the diagonal line parallel to the second direction in the photoelectric conversion unit 17. Means. Further, the half pitch may mean half of the repetition length of the photoelectric conversion unit 17 in the first direction and half of the repetition length of the photoelectric conversion unit 17 in the second direction. Since the plurality of photoelectric conversion units 17-R and 17-B are provided with a half-pitch shift with respect to the plurality of photoelectric conversion units 17-G, the plurality of photoelectric conversion units 17-G provided in the first and second directions are provided. A plurality of photoelectric conversion units 17-R and 17-B are provided in a plurality of gaps therebetween.

  The plurality of photoelectric conversion units 17-G are continuously provided in the first direction to form a linear shape. Similarly, the plurality of photoelectric conversion units 17-B and 17-R are continuously provided in the first direction to form a linear shape. The linear shape is provided in the second direction in the order of the plurality of photoelectric conversion units 17-G, 17-R, 17-G, and 17-B. A pattern composed of the set of the linear shapes 17-G, 17-R, 17-G, and 17-B is repeatedly provided in the second direction.

  Each photoelectric conversion unit 17 has an opening region. Each photoelectric conversion unit 17 may have an opening region and a light shielding region. In this example, each of the plurality of photoelectric conversion units 17 has an opening region and a light shielding region. Moreover, in this example, the area of the opening area | region and light-shielding area | region in each photoelectric conversion part 17 is substantially equal. In FIG. 3, the light shielding region is indicated by hatching. In the light shielding region, incident light is not used for photoelectric conversion. On the other hand, in the opening region, incident light is used for photoelectric conversion.

  The imaging element 200 of this example is a back-illuminated imaging element 200 in which a photodiode is provided on the microlens 12 side of the wiring layer in the photoelectric conversion unit 17. However, in the photoelectric conversion unit 17, the surface irradiation type imaging device 200 in which the wiring layer is provided closer to the microlens 12 than the photodiode may be used. In the surface irradiation type, incident light is partially blocked by the wiring layer. In the back-illuminated type, incident light is not partially blocked by the wiring layer unlike the front-illuminated type, so that the incident light conversion efficiency can be made higher than that of the front-illuminated type.

  In this example, the photoelectric conversion unit 17-G is divided into two regions in the second direction. One of the two parts is an opening area, and the other is a light shielding area. In the photoelectric conversion units 17-G adjacent in the first direction, the opening regions and the light shielding regions are alternately switched.

  In this example, the photoelectric conversion units 17-B and 17-R are divided into two regions in the first direction. One of the two parts is an opening area, and the other is a light shielding area. In the photoelectric conversion units 17-B and 17-R adjacent in the second direction, the opening region and the light shielding region are alternately switched.

  The light shielding regions of the two photoelectric conversion units 17-G adjacent in the second direction and the light shielding regions of the two photoelectric conversion units 17-R adjacent in the first direction form a square light shielding region adjacent to each other. . The square light-shielding region is a square having sides parallel to the first direction and the second direction.

  Further, the light shielding regions of the two photoelectric conversion units 17-G adjacent in the second direction and the light shielding regions of the two photoelectric conversion units 17-B adjacent in the first direction form a light shielding region adjacent to each other. . The square light-shielding region is a square having sides parallel to the first direction and the second direction.

  The opening area of the photoelectric conversion unit 17 is used for focus detection. In a plurality of photoelectric conversion units 17-G adjacent in the first direction, a plurality of photoelectric conversion units 17-G having an opening region in the second direction and a plurality of photoelectric conversion units 17 having an opening region in the opposite direction to the second direction. -G is used for focus detection. In this example, the plurality of photoelectric conversion units 17-G have light shielding regions alternately in the first direction and the reverse direction of the first direction, and alternately light shielding regions in the reverse direction of the second direction and the second direction. Have Therefore, the plurality of photoelectric conversion units 17-G can be used for focus detection in the first direction and the second direction. Thereby, compared with the case where the photoelectric conversion part 17 which has a light-shielding area | region is provided continuously only in any one of a 1st direction or a 2nd direction, a focus detection precision can be improved.

  The plurality of photoelectric conversion units 17-B and 17-R can be used for focus detection in the first direction. As a modification, the image sensor 200 may employ a pixel division method in which one photoelectric conversion unit 17 has a pair of opening regions in either the first direction or the second direction. The pair of opening regions are provided separately from each other. In addition, charges can be read independently from each other from photodiodes provided corresponding to the pair of opening regions. Focus detection can also be performed by using a plurality of photoelectric conversion units 17 as a pixel division method.

  The signal processing unit 32 in the filter processing mode first performs directionality determination in the determination / interpolation region 40. The directionality determination is to determine the continuity of output signals in at least one direction of the first direction and the second direction based on output signals from the plurality of photoelectric conversion units 17. In the determination / interpolation region 40, when the difference value of the output signals of the plurality of photoelectric conversion units 17 provided continuously in the first direction or the second direction is equal to or less than a predetermined threshold value, the signal processing unit 32 It is determined that there is continuity in one direction or the second direction. For example, in the determination / interpolation region 40, when the difference value between the output signals of four photoelectric conversion units 17-R adjacent in the first direction is equal to or less than a predetermined threshold value, the signal processing unit 32 moves in the first direction. Judge that there is continuity.

  The signal processing unit 32 in the filter processing mode performs interpolation processing in the determination / interpolation region 40 after the directionality determination. The interpolation processing is applied by adjusting the filter for each pixel based on the directionality determination. A pixel value is calculated by applying a filter. Note that applying a filter to a pixel means that the output signal of the photoelectric conversion unit 17 is filtered. The output signal after the filter processing is referred to as a pixel value. Each pixel value corresponds to each of a plurality of pixels in the finally output image.

  For example, when the signal processing unit 32 determines that there is continuity in the first direction, the signal processing unit 32 applies a filter for continuity in the first direction. Similarly, when it is determined that there is continuity in the second direction, a filter for continuity in the second direction is applied. In this example, the first direction continuity filter is a filter in the case where there is continuity in the vertical direction in FIGS. Moreover, the filter for 2nd direction continuity is a filter in case there exists continuity in the 2nd direction in FIGS.

  The filter in the filter processing mode refers to an operation on the output signals of the plurality of photoelectric conversion units 17 for calculating one pixel value of the output image. The filter of this example has a smoothing filter and a sharpening filter. The smoothing filter is an operation that smoothly and continuously makes adjacent pixel values in the output image. In this example, a Gaussian filter that outputs a weighted specific position is used as a smoothing filter. The sharpening filter is an operation that emphasizes light and darkness or edges in an output image. In this example, a Laplacian filter is used as the sharpening filter. Note that a Butterworth type filter realized by an IIR (Infinite Impulse Response) filter or the like may be used.

  The shape of the determination / interpolation area 40 is determined in advance. The determination / interpolation area 40 of this example includes at least three rows parallel to the second direction and four columns of photoelectric conversion units 17-G parallel to the first direction, and two rows parallel to the second direction and the first direction. 3 columns of photoelectric conversion units 17-B, and 4 rows parallel to the second direction and 2 columns of photoelectric conversion units 17-R parallel to the first direction. The determination / interpolation area 40 is centered on the position of the boundary between the two photoelectric conversion units 17-G and the two photoelectric conversion units 17-B adjacent to each other. The center 42 is located at the boundary between the two photoelectric conversion units 17-G in the second row among the three rows of photoelectric conversion units 17-G parallel to the second direction. The center 42 is indicated by a number surrounded by a circle. The plurality of photoelectric conversion units 17 provided in the determination / interpolation region 40 are symmetrical with respect to a straight line passing through the center 42 and parallel to the first direction and a straight line parallel to the second direction. The shape of the determination / interpolation area 40 is not limited to the above-described form. The shape of the determination / interpolation region 40 can be any shape as long as the directionality determination and the interpolation processing can be applied.

  In the conventional method described in Patent Document 1, the output signal for green (G) corresponding to one pixel is obtained by adding the values of the output signals of the two photoelectric conversion units 17-G. However, since the output signal for green (G) corresponding to one pixel is obtained using the two photoelectric conversion units 17-G, the pixel density in the image is reduced to half. On the other hand, in this example, the position of the boundary between the two photoelectric conversion units 17-G and the two photoelectric conversion units 17-B, and the two photoelectric conversion units 17-B and the two photoelectric conversion units 17-R. The pixel values of green (G), blue (B), and red (R) are respectively generated using the output signals of neighboring pixels at the position of the boundary. Thereby, the pixel density in the image can be improved as compared with the conventional method.

  FIG. 4 is a diagram illustrating a case where there is continuity in the first direction in the determination / interpolation region 40. In FIG. 4, in order to make the drawing easy to see, the description of the microlens 12, the aperture region, and the light shielding region described in FIG. 3 is omitted. In addition, G shows the photoelectric conversion part 17-G for green simply. The number following G is a coordinate number described for explanation. Thereafter, the type and position of the photoelectric conversion unit 17 are specified by one alphabetic character and the coordinate number. B represents the photoelectric conversion unit 17-B, and R represents the photoelectric conversion unit 17-R.

  (First Directionality Determination) The signal processing unit 32 may calculate the difference between the output signals of the photoelectric conversion units 17 adjacent to each other in the first direction in the determination / interpolation region 40. Thereby, the signal processing unit 32 determines continuity in the first direction. In this example, the signal processing unit 32 calculates a difference between R output signals adjacent to each other in the first direction. That is, the signal processing unit 32 outputs R output signals adjacent to each other for at least one of the region 46-1 including R11, R21, R31, and R41 and the region 46-2 including R12, R22, R32, and R42. Calculate the difference.

  Assume that the output signal of [R11, R21, R31, R41] is [90, 90, 91, 91] in the region 46-1. In this case, the difference vectors [R11-R21, R21-R31, R31-R41] of the R output signals adjacent to each other are [0, -1, 0]. Further, it is assumed that the output signal of [R11, R21, R31, R41] is [90, 10, 91, 10] in the region 46-1. In this case, the difference vector of the R output signals adjacent to each other is [80, −81, 81]. For example, the signal processing unit 32 may determine that there is continuity when the absolute value of each component of the difference value vector is 10 or less. In this example, the signal processing unit 32 determines that the difference vector [0, −1, 0] is continuous in the first direction, and the difference vector [80, −81, 81] in the first direction. It is determined that there is no continuity. Note that the threshold value of the component of the difference vector that is determined to have continuity may be appropriately determined by program correction.

(Interpolation processing when there is first directionality: g ₃₃ ) The signal processing unit 32 performs interpolation processing in the determination / interpolation region 40 after the directionality determination. When the signal processing unit 32 determines that there is directionality in the first direction, the signal processing unit 32 applies a smoothing filter to a row × b column G output signals in the determination / interpolation region 40. Note that a is a natural number larger than b, and in this example, a = 3 and b = 2.

The signal processing unit 32 of this example applies a Gaussian filter to 3 rows × 2 columns of G output signals (Equation 1) in the determination / interpolation region 40. (Equation 2) is an example of a Gaussian filter.

Then, the signal processing unit 32 calculates the convolution values of (Equation 1) and (Equation 2). The convolution operation is calculated by the following (Equation 3) for a matrix of M rows and N columns. M, N, m, and n are natural numbers. However, in (Equation 3), when M is an even number, m ₁ = −m / 2, m ₂ = m / 2, and m = 0 is excluded in the calculation of Σ. When M is an odd number, m ₁ = − (m−1) / 2 and m ₂ = (m−1) / 2. Further, in (Equation 3), when N is an even number, n ₁ = −n / 2 and n ₂ = n / 2, and n = 0 is excluded in the calculation of Σ. When N is an odd number, n ₁ = − (n−1) / 2 and n ₂ = (n−1) / 2.

In this example, in (Equation 3), the function h = (Equation 2), the function f = (Equation 1), M = 3, and N = 2. In addition, m ₁ = −1 and m ₂ = 1. n ₁ = −1, n ₂ = 1, and n = 0 is excluded in the calculation of Σ. Here, m ₂ designates the first row of (Equation 1) and (Equation 2), m ₁ designates the third row of (Equation 1) and (Equation 2), and n ₂ represents (Equation 1). And the first column of (Equation 2) is designated, and n ₁ designates the second column of (Equation 1) and (Equation 2). As a result, g _33-G to which the Gaussian filter is applied becomes g _33-G = (1 × G ₁₂ + 2 × G ₂₂ + 1 × G ₃₂ + 1 × G ₁₃ + 2 × G ₂₃ + 1 × G ₃₃ ) / 8.

The signal processing unit 32 applies a sharpening filter to c rows × d columns of R output signals in the determination / interpolation region 40. Note that c is a natural number larger than d, and in this example, c = 4 and d = 2. The signal processing unit 32 in this example applies a Laplacian filter to 4 rows × 2 columns of R output signals (Equation 4) in the determination / interpolation region 40. (Equation 5) is an example of a Laplacian filter.

Then, the signal processing unit 32 calculates the convolution values of (Equation 4) and (Equation 5). The convolution operation is the same as in (Equation 3). In this example, in (Equation 3), the function h = (Equation 5), the function f = (Equation 4), M = 4, and N = 2. Further, m ₁ = −2, m ₂ = 2, and n = 0 is excluded in the calculation of Σ. n ₁ = −1, n ₂ = 1, and n = 0 is excluded in the calculation of Σ. Here, m ₂ designates the first row of (Equation 4) and (Equation 5), m ₁ designates the fourth row of (Equation 4) and (Equation 5), and n ₂ represents (Equation 4). And the first column of (Equation 5) is designated, and n ₁ designates the second column of (Equation 4) and (Equation 5). As a result, g _33-L to which the Laplacian filter is applied becomes g _33-L = (− 1 × R ₁₁ + 1 × R ₂₁ + 1 × R ₃₁ −1 × R ₄₁ −1 × R ₁₂ + 1 × R ₂₂ + 1 × R ₃₂ −1 × R ₄₂ ) / 8.

The signal processing unit 32 calculates the G pixel value corresponding to the position of the center 42 by taking the sum of the value to which the smoothing filter is applied and the value to which the sharpening filter is applied. That is, the sum of g _33-G and g _33-L is the corrected pixel value g ₃₃ at the center 42. Thus in this example, in order to calculate the pixel value of g _33, also added Laplacian of R in addition to the Gaussian G. Thus, as compared with the case where the g _33-G alone g _33, can be calculated more accurately pixel values.

(Interpolation processing when there is first directionality: b ₃₃ , r ₃₃ ) The signal processing unit 32 calculates a pixel value g ₃₃ and calculates b ₃₃ obtained by correcting the output signal of B. The signal processing unit 32 calculates a value obtained by applying a smoothing filter to k rows × l columns of B output signals in the determination / interpolation region 40 as a B pixel value corresponding to the position of the center 42. In this example, k is a natural number larger than l, and k = 2 and l = 1. The signal processing unit 32 of this example applies a Gaussian filter to 2 rows × 1 columns of B output signals (Equation 6) in the determination / interpolation region 40. Then, the convolution of (Equation 6) and (Equation 7) is calculated as a pixel value b ₃₃ after correction. (Equation 7) is an example of a Gaussian filter.

In this example, in (Equation 3), the function h = (Equation 7), the function f = (Equation 6), M = 2, and N = 1. Further, m ₁ = −1, m ₂ = 1, and n = 0 is excluded in the calculation of Σ. n ₁ = n ₂ = 0. Here, m ₂ designates the first row of (Equation 6) and (Equation 7), m ₁ designates the second row of (Equation 6) and (Equation 7), and n ₁ and n ₂ represent ( Assume that the first column of (Equation 6) and (Equation 7) is specified. As a result, b ₃₃ to which the Gaussian filter is applied becomes b ₃₃ = (1 × B ₁₂ + 1 × B ₂₂ ) / 2.

The signal processing unit 32 further calculates r ₃₃ obtained by correcting the R output signal. The signal processing unit 32 calculates a value obtained by applying a smoothing filter to p rows × q columns of R output signals in the determination / interpolation region 40 as an R pixel value corresponding to the position of the center 42. In this example, p is a natural number larger than q, and p = 4 and q = 2. The signal processing unit 32 of this example applies a Gaussian filter to 4 rows × 2 columns of R output signals (Equation 8) in the determination / interpolation region 40. The signal processing unit 32 calculates a pixel value _{r 33} after correcting the convolution of equation (8) and (9). (Equation 9) is an example of a Gaussian filter.

In this example, in (Equation 3), the function h = (Equation 9), the function f = (Equation 8), M = 4, and N = 2. m ₁ = −2, m ₂ = 2, and n = 0 is excluded in the calculation of Σ. n ₁ = −1, n ₂ = 1, and n = 0 is excluded in the calculation of Σ. Here, m ₂ designates the first row of (Equation 8) and (Equation 9), m ₁ designates the fourth row of (Equation 8) and (Equation 9), and n ₂ represents (Equation 8). And the first column of (Equation 9) is designated, and n ₁ designates the second column of (Equation 8) and (Equation 9). Accordingly, r ₃₃ to which the Gaussian filter is applied is expressed as follows: r ₃₃ = (1 × R ₁₁ + 3 × R ₂₁ + 3 × R ₃₁ + 1 × R ₄₁ + 1 × R ₁₂ + 3 × R ₂₂ + 3 × R ₃₂ + 1 × R ₄₂ ) / 16

  FIG. 5 is a diagram illustrating a case where there is continuity in the second direction in the determination / interpolation region 40. As in FIG. 4, the description of the microlens 12, the opening area, and the light shielding area is omitted.

  (Second Directionality Determination) The signal processing unit 32 may calculate a difference between output signals of the photoelectric conversion units 17 adjacent to each other in the second direction in the determination / interpolation region 40. Thereby, the signal processing unit 32 determines continuity in the second direction. In this example, the signal processing unit 32 calculates the difference between the B output signals adjacent to each other in the second direction. That is, the signal processing unit 32 calculates the difference between the adjacent B output signals for at least one of the region 48-1 including B11, B12, and B13 and the region 48-2 including B21, B22, and B23. .

  Assume that the output signal of [B11, B12, B13] is [90, 91, 91] in the region 48-1. In this case, the difference vector [B11−B11, B12−B13] of the R output signals adjacent to each other is [−1, 0]. Further, it is assumed that the output signal of [B11, B12, B13] is [90, 10, 91] in the region 48-1. In this case, the difference vector of the B output signals adjacent to each other is [80, −81]. For example, the signal processing unit 32 may determine that there is continuity when the absolute value of each component of the difference value vector is 10 or less. In this example, the signal processing unit 32 determines that the difference vector [−1, 0] has continuity in the second direction, and the difference vector [80, −81] has no continuity in the second direction. Is determined. Note that the threshold value of the component of the difference vector that is determined to have continuity may be appropriately determined by program correction.

(Interpolation process when there is second directionality: g ₃₃ ) The signal processing unit 32 performs an interpolation process in the determination / interpolation region 40 after the directionality determination. When determining that there is directionality in the second direction, the signal processing unit 32 applies a smoothing filter to r rows × s columns of G output signals in the determination / interpolation region 40. Note that r is a natural number smaller than s, and in this example, r = 3 and s = 4. Thereby, the pixel value g ₃₃ at the center 42 of the determination / interpolation area 40 is calculated.

The signal processing unit 32 of this example applies a Gaussian filter to 3 rows × 4 columns of G output signals (Equation 10) in the determination / interpolation region 40. (Equation 11) is an example of a Gaussian filter.

Then, the signal processing unit 32 calculates the convolution values of (Equation 10) and (Equation 11). In this example, in (Equation 3), the function h = (Equation 11), the function f = (Equation 10), M = 3, and N = 4. In addition, m ₁ = −1 and m ₂ = 1. n ₁ = −2, n ₂ = 2, and n = 0 is excluded in the calculation of Σ. Here, m ₂ designates the first row of (Equation 10) and (Equation 11), m ₁ designates the third row of (Equation 10) and (Equation 11), and n ₂ represents (Equation 10). And the first column of (Equation 11) is designated, and n ₁ designates the fourth column of (Equation 10) and (Equation 11). As a result, g _33-G to which the Gaussian filter is applied becomes g _33-G = (1 × G ₁₁ + 2 × G ₂₁ + 1 × G ₃₁ + 3 × G ₁₂ + 6 × G ₂₂ + 3 × G ₃₂ + 3 × G ₁₃ + 6 × G ₂₃ + 3 × G ₃₃ + 1 × G ₁₄ + 2 × G ₂₄ + 1 × G ₃₄ ) / 32.

In addition, the signal processing unit 32 applies a sharpening filter to t row × u column B output signals in the determination / interpolation region 40. Note that t is a natural number smaller than u, and in this example, t = 2 and u = 3. The signal processing unit 32 of this example applies a Laplacian filter to 2 rows × 3 columns of B output signals (Equation 12) in the determination / interpolation region 40. (Equation 13) is an example of a Laplacian filter.

Then, the signal processing unit 32 calculates the convolution values of (Equation 12) and (Equation 13). In this example, in (Equation 3), the function h = (Equation 13), the function f = (Equation 12), M = 2, and N = 3. Further, m ₁ = −1, m ₂ = 1, and n = 0 is excluded in the calculation of Σ. n ₁ = -1, n ₂ = 1. Here, m ₂ designates the first row of (Equation 12) and (Equation 13), m ₁ designates the second row of (Equation 12) and (Equation 13), and n ₂ represents (Equation 12). And the first column of (Equation 13) is designated, and n ₁ designates the third column of (Equation 12) and (Equation 13). Thereby, g _33-L to which the Laplacian filter is applied is g _33-L = (− 1 × B ₁₁ −1 × B ₂₁ + 2 × B ₁₂ + 2 × B ₂₂ −1 × B ₁₃ −1 × B ₂₃ ) / 8

The G pixel value corresponding to the position of the center 42 is calculated by taking the sum of the value to which the smoothing filter is applied and the value to which the sharpening filter is applied. That is, the sum of g _33-G and g _33-L is the corrected pixel value g ₃₃ at the center 42. Thus, in this example, in order to calculate the pixel value of g ₃₃ , B Laplacian is added in addition to G Gaussian. Therefore, as compared with the case where the g _33-G alone g _33, the image can be more sharpened.

(Interpolation processing when there is a second direction: b ₃₃ , r ₃₃ ) The signal processing unit 32 calculates g ₃₃ obtained by correcting the G output signal, and calculates b ₃₃ obtained by correcting the B output signal. . The signal processing unit 32 calculates a value obtained by applying a smoothing filter to v row × w column B output signals in the determination / interpolation region 40 as a B pixel value corresponding to the position of the center 42. In this example, v is a natural number smaller than w, and v = 2 and w = 3. The signal processing unit 32 of this example applies a Gaussian filter to 2 rows × 3 columns of B output signals (Expression 14) in the determination / interpolation region 40. The signal processing unit 32 calculates the (number 14) and the pixel value _{b 33} values of the convolution of the corrected (several 15). (Equation 15) is an example of a Gaussian filter.

In this example, in (Equation 3), the function h = (Equation 15), the function f = (Equation 14), M = 2, and N = 3. Therefore, m ₁ = −1, m ₂ = 1, and m = 0 is excluded in the calculation of Σ. Further, n ₁ = −1 and n ₂ = 1. Here, m ₂ designates the first row of (Equation 14) and (Equation 15), m ₁ designates the second row of (Equation 14) and (Equation 15), and n ₂ represents (Equation 14). And the first column of (Equation 15) is designated, and n ₁ designates the third column of (Equation 14) and (Equation 15). As a result, b ₃₃ to which the Gaussian filter is applied is b ₃₃ = (1 × B ₁₁ + 1 × B ₂₁ + 2 × B ₁₂ + 2 × B ₂₂ + 1 × B ₁₃ + 1 × B ₂₃ ) / 8.

Similarly, the signal processing unit 32 calculates a value obtained by applying a smoothing filter to the x row × y column R output signals in the determination / interpolation region 40 as an R pixel value corresponding to the position of the center 42. . In this example, x and y are equal natural numbers, and x = y = 2. The signal processing unit 32 of this example applies a Gaussian filter to 2 rows × 2 columns of R output signals (Expression 16) in the determination / interpolation region 40. The signal processing unit 32 calculates the value of the convolution pixel values _{r 33} after correction of Equation 16 and equation (17). (Equation 17) is an example of a Gaussian filter.

In this example, in (Equation 3), the function h = (Equation 17), the function f = (Equation 16), M = 2, and N = 2. Therefore, m ₁ = −1, m ₂ = 1, and m = 0 is excluded in the calculation of Σ. Further, n ₁ = −1, n ₂ = 1, and n = 0 is excluded in the calculation of Σ. Here, m ₂ designates the first row of (Equation 16) and (Equation 17), m ₁ designates the second row of (Equation 16) and (Equation 17), and n ₂ represents (Equation 16). And the first column of (Equation 17) is designated, and n ₁ designates the second column of (Equation 16) and (Equation 17). As a result, r ₃₃ to which the Gaussian filter is applied becomes r ₃₃ = (1 × R ₁₂ + 1 × R ₃₁ + 1 × R ₂₂ + 1 × R ₃₂ ) / 4.

  FIG. 6 is a diagram showing the determination / interpolation area 41. The determination / interpolation area 41 is an area obtained by moving the determination / interpolation area 40 by one pitch in the first direction and the second direction. One pitch is twice as long as a half pitch. The determination / interpolation area 41 is centered on the position of the boundary between two G and two R adjacent to each other. The determination / interpolation area 41 includes at least 3 rows parallel to the second direction and 4 columns G parallel to the first direction, 4 rows parallel to the second direction, and 2 parallel to the first direction. Column B and two rows parallel to the second direction and three columns R parallel to the first direction are included. The arrangement of B and R in the determination / interpolation area 41 corresponds to the arrangement of R and B in the determination / interpolation area 40, respectively.

The signal processing unit 32 performs directionality determination and interpolation processing in the determination / interpolation region 41 as in the processing in the determination / interpolation region 40. Pixel values g ₄₄ , b ₄₄ and r ₄₄ that can be reached at the position of the center ₄₄ are generated by the interpolation processing.

(Interpolation processing when there is first directionality: g ₄₄ ) The signal processing unit 32 performs interpolation processing in the determination / interpolation region 41 after determining that there is continuity in the first direction. When the signal processing unit 32 determines that there is directionality in the first direction, the signal processing unit 32 applies a smoothing filter to a row × b column G output signals in the determination / interpolation region 41. Note that a is a natural number larger than b, and in this example, a = 3 and b = 2. As a result, the pixel value g ₄₄ at the center 44 of the determination / interpolation area 41 is calculated.

The signal processing unit 32 of this example applies a Gaussian filter to 3 rows × 2 columns of G output signals (Equation 18) in the determination / interpolation region 41. (Equation 19) is an example of a Gaussian filter.

Then, the signal processing unit 32 calculates the convolution values of (Equation 18) and (Equation 19). In this example, in (Equation 3), the function h = (Equation 19), the function f = (Equation 18), M = 3, and N = 2. In addition, m ₁ = −1 and m ₂ = 1. n ₁ = −1, n ₂ = 1, and n = 0 is excluded in the calculation of Σ. Here, m ₂ designates the first row of (Equation 18) and (Equation 19), m ₁ designates the third row of (Equation 18) and (Equation 19), and n ₂ represents (Equation 18). And the first column of (Equation 19) is designated, and n ₁ designates the second column of (Equation 18) and (Equation 19). Thus, g _44-G to which the Gaussian filter is applied becomes g _44-G = (1 × G ₂₃ + 2 × G ₃₃ + 1 × G ₄₃ + 1 × G ₂₄ + 2 × G ₃₄ + 1 × G ₄₄ ) / 8.

In addition, the signal processing unit 32 applies a sharpening filter to the c row × d column B output signals in the determination / interpolation region 41. Note that c is a natural number larger than d, and in this example, c = 4 and d = 2. The signal processing unit 32 in this example applies a Laplacian filter to 4 rows × 2 columns of B output signals (Equation 20) in the determination / interpolation region 41. (Equation 21) is an example of a Laplacian filter.

Then, the signal processing unit 32 calculates the convolution values of (Equation 20) and (Equation 21). In this example, in (Equation 3), the function h = (Equation 21), the function f = (Equation 20), M = 4, and N = 2. Further, m ₁ = −2, m ₂ = 2, and n = 0 is excluded in the calculation of Σ. n ₁ = −1, n ₂ = 1, and n = 0 is excluded in the calculation of Σ. Here, m ₂ designates the first row of (Equation 20) and (Equation 21), m ₁ designates the fourth row of (Equation 20) and (Equation 21), and n ₂ represents (Equation 20). And the first column of (Equation 21) is designated, and n ₁ designates the second column of (Equation 20) and (Equation 21). As a result, g _{44 -L} to which the Laplacian filter is applied becomes g _{44 -L} = (− 1 × B ₁₂ + 1 × B ₂₂ + 1 × B ₃₂ −1 × B ₄₂ −1 × B ₁₃ + 1 × B ₂₃ + 1 × B ₃₃ −1 × B ₄₃ ) / 8.

The G pixel value corresponding to the position of the center 44 is calculated by taking the sum of the value to which the smoothing filter is applied and the value to which the sharpening filter is applied. That is, the sum of g _{44 -G} and g _{44 -L} is the corrected pixel value g ₄₄ at the center 44.

(Interpolation process when there is first directionality: b ₄₄ , r ₄₄ ) The signal processing unit 32 calculates a pixel value g ₄₄ and calculates b ₄₄ obtained by correcting the B pixel value. The signal processing unit 32 calculates a value obtained by applying a smoothing filter to k rows × l columns of B output signals in the determination / interpolation region 41 as a B pixel value corresponding to the position of the center 42. In this example, k is a natural number larger than l, and k = 4 and l = 2. The signal processing unit 32 of this example applies a Gaussian filter to 4 rows × 2 columns of B output signals (Equation 22) in the determination / interpolation region 41. The signal processing unit 32 calculates a pixel value _{b 44} after the correction values of the convolution of equation (22) and (Equation 23). (Equation 23) is an example of a Gaussian filter.

In this example, in (Equation 3), the function h = (Equation 23), the function f = (Equation 22), M = 4, and N = 2. m ₁ = −2, m ₂ = 2, and n = 0 is excluded in the calculation of Σ. n ₁ = 1 and n ₂ = 1, and n = 0 is excluded in the calculation of Σ. Here, m ₂ designates the first row of (Equation 22) and (Equation 23), m ₁ designates the fourth row of (Equation 22) and (Equation 23), and n ₂ represents (Equation 22). And the first column of (Equation 23) is designated, and n ₁ designates the second column of (Equation 22) and (Equation 23). Thus, b ₄₄ to which the Gaussian filter is applied is b ₄₄ = (1 × B ₁₂ + 3 × B ₂₂ + 3 × B ₃₂ + 1 × B ₄₂ + 1 × B ₁₃ + 3 × B ₂₃ + 3 × B ₃₃ + 1 × B ₄₃ ) / 16

The signal processing unit 32 further calculates the r ₄₄ obtained by correcting the pixel values of R. The signal processing unit 32 calculates a value obtained by applying a smoothing filter to p rows × q columns of R output signals in the determination / interpolation region 41 as an R pixel value corresponding to the position of the center 42. In this example, p is a natural number larger than q, and p = 2 and q = 1. The signal processing unit 32 of this example applies a Gaussian filter to 2 rows × 1 columns of R output signals (Equation 24) in the determination / interpolation region 41. The signal processing unit 32 calculates a pixel value _{r 44} after the correction values of the convolution of equation (24) and (Equation 25). Note that (Equation 25) is an example of a Gaussian filter.

In this example, in (Equation 3), the function h = (Equation 25), the function f = (Equation 24), M = 2, and N = 1. Further, m ₁ = −1, m ₂ = 1, and n = 0 is excluded in the calculation of Σ. n ₁ = n ₂ = 0. Here, m ₂ specifies the first row of (Equation 24) and (Equation 25), m ₁ specifies the second row of (Equation 24) and (Equation 25), and n ₁ and n ₂ are ( Assume that the first column of (Equation 24) and (Equation 25) is designated. Accordingly, r ₄₄ to which the Gaussian filter is applied becomes r ₄₄ = (1 × R ₃₂ + 1 × R ₄₂ ) / 2.

(Interpolation process when there is a second direction: g ₄₄ ) The signal processing unit 32 performs an interpolation process in the determination / interpolation region 41 after determining that there is continuity in the second direction. When determining that there is directionality in the second direction, the signal processing unit 32 applies a smoothing filter to r rows × s columns of G output signals in the determination / interpolation region 41. Note that r is a natural number smaller than s, and in this example, r = 3 and s = 4. As a result, the pixel value g ₄₄ at the center 44 of the determination / interpolation area 40 is calculated.

The signal processing unit 32 of this example applies a Gaussian filter to 3 rows × 4 columns of G output signals (Equation 26) in the determination / interpolation region 41. (Equation 27) is an example of a Gaussian filter.

Then, the signal processing unit 32 calculates the convolution values of (Equation 26) and (Equation 27). In this example, in (Expression 3), the function h = (Expression 27), the function f = (Expression 26), M = 3, and N = 4. In addition, m ₁ = −1 and m ₂ = 1. n ₁ = −2, n ₂ = 2, and n = 0 is excluded in the calculation of Σ. Here, m ₂ designates the first row of (Equation 26) and (Equation 27), m ₁ designates the third row of (Equation 26) and (Equation 27), and n ₂ represents (Equation 26). And the first column of (Equation 27) is designated, and n ₁ designates the fourth column of (Equation 26) and (Equation 27). As a result, g _44-G to which the Gaussian filter is applied becomes g _44-G = (1 × G ₂₂ + 2 × G ₃₂ + 1 × G ₄₂ + 3 × G ₂₃ + 6 × G ₃₃ + 3 × G ₄₃ + 3 × G ₂₄ + 6 × G ₃₄ + 3 × G ₄₄ + 1 × G ₂₅ + 2 × G ₃₅ + 1 × G ₄₅ ) / 32.

Further, the signal processing unit 32 applies a sharpening filter to t rows × u columns R output signals in the determination / interpolation region 41. Note that t is a natural number smaller than u, and in this example, t = 2 and u = 3. The signal processing unit 32 of this example applies a Laplacian filter to 2 rows × 3 columns of R output signals (Equation 28) in the determination / interpolation region 41. (Equation 29) is an example of a Laplacian filter.

Then, the signal processing unit 32 calculates the convolution values of (Equation 28) and (Equation 29). In this example, in (Equation 3), the function h = (Equation 29), the function f = (Equation 28), M = 2, and N = 3. Further, m ₁ = −1, m ₂ = 1, and n = 0 is excluded in the calculation of Σ. n ₁ = -1, n ₂ = 1. Here, m ₂ designates the first row of (Equation 28) and (Equation 29), m ₁ designates the second row of (Equation 28) and (Equation 29), and n ₂ represents (Equation 28). And the first column of (Equation 29) is designated, and n ₁ designates the third column of (Equation 28) and (Equation 29). Thus, g _44-L to which the Laplacian filter is applied is g _44-L = (− 1 × R ₃₁ −1 × R ₄₁ + 2 × R ₃₂ + 2 × R ₄₂ −1 × R ₃₃ −1 × R ₄₃ ) / 8

The G pixel value corresponding to the position of the center 44 is calculated by taking the sum of the value to which the smoothing filter is applied and the value to which the sharpening filter is applied. That is, the sum of g _{44 -G} and g _{44 -L} is the corrected pixel value g ₄₄ at the center 44.

(Interpolation process when there is second direction: b ₄₄ , r ₄₄ ) The signal processing unit 32 calculates a pixel value g ₄₄ and also calculates a pixel value r ₄₄ obtained by correcting the R output signal. The signal processing unit 32 calculates a value obtained by applying a smoothing filter to v row × w column R output signals in the determination / interpolation region 41 as an R pixel value corresponding to the position of the center 42. In this example, v is a natural number smaller than w, and v = 2 and w = 3. The signal processing unit 32 of this example applies a Gaussian filter to 2 rows × 3 columns of R output signals (Equation 30) in the determination / interpolation region 41. The signal processing unit 32 calculates a pixel value _{r 44} after the correction values of the convolution of equation (30) and (Equation 31). (Equation 31) is an example of a Gaussian filter.

In this example, in (Equation 3), M = 2 and N = 3. Therefore, m ₁ = −1, m ₂ = 1, and m = 0 is excluded in the calculation of Σ. Further, n ₁ = −1 and n ₂ = 1. Here, m ₂ designates the first row of (Equation 30) and (Equation 31), m ₁ designates the second row of (Equation 30) and (Equation 31), and n ₂ represents (Equation 30). And the first column of (Equation 31) is designated, and n ₁ designates the third column of (Equation 30) and (Equation 31). As a result, r ₄₄ to which the Gaussian filter is applied becomes r ₄₄ = (1 × R ₃₁ + 1 × R ₄₁ + 2 × R ₃₂ + 2 × R ₄₂ + 1 × R ₃₃ + 1 × R ₄₃ ) / 8.

Similarly, the signal processing unit 32 calculates a value obtained by applying a smoothing filter to x row × y column B output signals in the determination / interpolation region 41 as a B pixel value corresponding to the position of the center 44. . In this example, x and y are equal natural numbers, and x = y = 2. The signal processing unit 32 of this example applies a Gaussian filter to 2 rows × 2 columns of B output signals (Expression 32) in the determination / interpolation region 41. The signal processing unit 32 calculates a pixel value _{b 44} after the correction values of the convolution of equation (32) and (Equation 33). Note that (Expression 33) is an example of a Gaussian filter.

In this example, in (Equation 3), M = 2 and N = 2. Therefore, m ₁ = −1, m ₂ = 1, and m = 0 is excluded in the calculation of Σ. Further, n ₁ = −1, n ₂ = 1, and n = 0 is excluded in the calculation of Σ. Here, m ₂ designates the first row of (Equation 32) and (Equation 33), m ₁ designates the second row of (Equation 32) and (Equation 33), and n ₂ represents (Equation 32). And the first column of (Equation 33) is designated, and n ₁ designates the second column of (Equation 32) and (Equation 33). As a result, b ₄₄ to which the Gaussian filter is applied is b ₄₄ = (1 × B ₂₂ + 1 × B ₃₂ + 1 × B ₂₃ + 1 × B ₃₃ ) / 4.

FIG. 7A is a diagram illustrating a correspondence relationship between each pixel in the determination / interpolation region 40 and the position of a pixel value generated by interpolation. As in FIG. 4, the description of the microlens 12, the opening area, and the light shielding area is omitted. FIG. 7B is a diagram illustrating pixel values generated by interpolation at positions corresponding to the determination / interpolation region 40. The pixel values g ₃₃ , b _33, and r ₃₃ interpolated at the position of the center 42 in FIG. 7A correspond to the pixel values g ₃₃ , b, and r at the position of the center 42 in FIG. 7B, respectively. Similarly, the pixel values g ₄₄ , b ₄₄ and r ₄₄ subjected to the interpolation processing at the position of the center 44 in FIG. 7A correspond to the pixel values g ₄₄ , b and r at the position of the center 44 in FIG. 7B, respectively.

Seven solid circles other than the center 42 and the center 44 in the determination / interpolation region 40 in FIG. 7A indicate the positions of the boundary between two Gs and two Rs adjacent to each other, and the two Gs and two Bs adjacent to each other. Indicates the position of the boundary. Pixel values generated by interpolation processing at the positions of the seven solid circles may be generated by performing the same processing as in the determination / interpolation regions 40 and 41. Pixel values g, b, and r generated by the interpolation processing at positions corresponding to the solid circles are shown in correspondence with FIG. 7B. For example, pixel values g ₂₂ , b _22, and r ₂₂ generated by interpolation processing at the positions of the boundaries of G 11, G 12, R 11, and R 21 are shown as g ₂₂ , b, and r in FIG. 7B. Note that the dotted circle in FIG. 7A corresponds to the dotted circle in FIG. 7B.

  In this example, the square photoelectric conversion units 17 are arranged in the first direction and the second direction. Pixel values r after interpolation using the output signals of R, G, and B at the boundary position between two G adjacent to each other in the second direction and two B or two R adjacent to each other in the first direction Generate g and b, respectively. Therefore, the resolution can be increased as compared with the conventional case of adding the output signals of adjacent pixels. Further, since the Laplacian filter is used in addition to the Gaussian filter, the image can be sharpened as compared with the case where only the Gaussian filter is used. Therefore, the resolution can be improved without sacrificing color reproducibility.

In this example, the method of interpolating pixel values for the determination / interpolation regions 40 and 41 having three rows of G parallel to the second direction and four columns of parallel to the first direction has been described. If the determination / interpolation regions 40 and 41 are expanded to include more G, B, and R, the accuracy of determining the directionality in the second direction and the first direction can be further improved. Further, when the pixel value is interpolated, the amount of calculation can be reduced by reducing the filter kernel size. For example, when there is continuity in the first direction, if the Gaussian filter for g ₃₃ is changed from 3 rows × 2 columns to 3 rows × 1 columns in this example, the amount of calculation can be reduced. Similarly, when there is continuity in the second direction, if the Gaussian filter for g ₃₃ is changed from 3 rows × 4 columns to 1 row × 4 columns in this example, the amount of calculation can be reduced. As a result, the processing speed can be improved.

  (Modification) In the first embodiment, the example in which all the photoelectric conversion units 17 have the light-shielding region and the opening region has been described. However, the G having the light shielding region and the opening region may be provided continuously only in the second direction. Further, the R having the light shielding region and the opening region may be provided continuously only in the first direction. The output signal of the photoelectric conversion unit 17 having the light shielding region is half of the output signal of the photoelectric conversion unit 17 having no light shielding region. Therefore, a process of doubling the pixel value of the photoelectric conversion unit 17 having the light shielding region in advance is performed. Thereby, directionality determination and interpolation processing can be applied in the same manner as the pixel values of other pixels.

  FIG. 8 is a flowchart 70 for explaining processing for switching between the addition mode and the filter processing mode in the second embodiment. When the user turns on the main body and turns on the imaging mode, the flow of this example is started (step 71). Thereafter, the signal processing unit 32 determines whether or not to perform live view display of an image capturing the subject (step 72). For example, the signal processing unit 32 may display an image in live view when the shutter release is not pressed. On the other hand, when the shutter release is pressed, the signal processing unit 32 may capture an image and store it in the recording unit 60 as data. The signal processing unit 32 includes a switching unit that switches whether to perform live view display.

  The switching unit selects an addition mode generated by adding signal values of two pixels adjacent in one of the first direction and the second direction when displaying an image in live view (step 73). On the other hand, the switching unit selects a filter processing mode in which a smoothing filter and a sharpening filter are applied when an image is stored as data in the recording unit 60 without performing live view display (step 74). In the filter processing mode, the signal processing unit 32 performs directionality determination and interpolation processing in the first embodiment.

  After the output display in the addition mode or the filter processing mode, the signal processing unit 32 determines whether or not the imaging mode is continued (step 75). If the imaging mode is not continued, the signal processing unit 32 ends the process. For example, when the user turns off the power of the single-lens reflex camera 400 or when the user turns off the imaging mode, the signal processing unit 32 ends the processing (step 76). When the imaging mode is continued, the signal processing unit 32 continues processing. For example, when the user maintains the imaging mode, the signal processing unit 32 continues the process (step 77). With this processing, in live view display, a coarse image is displayed in the addition mode, so that the processing burden on the signal processing unit 32 can be reduced. Therefore, it is possible to prevent the display from being interrupted when the moving subject is continuously displayed. On the other hand, when an image is stored as data, an image sharper than live view display can be recorded as data.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above embodiment. It is apparent from the description of the scope of claims that embodiments with such changes or improvements can be included in the technical scope of the present invention.

  The order of execution of each process such as operations, procedures, steps, and stages in the apparatus, system, program, and method shown in the claims, the description, and the drawings is particularly “before” or “prior to”. It should be noted that the output can be realized in any order unless the output of the previous process is used in the subsequent process. Even if the operation flow in the claims, the description, and the drawings is described using “first”, “next”, etc. for convenience, it means that it is essential to carry out in this order. It is not a thing.

10 imaging chip, 12 microlens, 15 bump, 17 photoelectric conversion unit, 30 signal processing chip, 32 signal processing unit, 34 AD conversion unit, 36 focus detection unit, 40 determination / interpolation region, 41 determination / interpolation region, 42 center 44 center, 46 region, 48 region, 60 recording unit, 70 flowchart, 200 imaging device, 340 shutter unit, 410 optical axis, 400 single-lens reflex camera, 500 lens unit, 550 lens mount, 600 camera body, 620 body substrate, 622 CPU, 624 Image processing unit, 625 Image processing ASIC, 634 Rear display unit, 650 Viewfinder, 652 Focus plate, 654 Pentaprism, 656 Viewfinder optical system, 660 Body mount, 670 Mirror box, 672 Mirror

Claims (1)

A plurality of pixels arranged two-dimensionally;
A signal processing unit that applies a smoothing filter and a sharpening filter to the output of the plurality of pixels to calculate each pixel value in the image, and
The plurality of pixels are:
A plurality of first pixels provided continuously in the first direction and photoelectrically converting incident light of the first wavelength;
An image pickup device comprising: a plurality of second pixels that are provided in the first direction so as to be shifted by a half pitch with respect to the plurality of first pixels and that photoelectrically convert incident light having a second wavelength.

Images