JP2009206552A - Image processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus that has few jaggies, is smooth, and can interpolate pixels having small number of false colors. <P>SOLUTION: Sections 12-15 for evaluating the presence or the absence and the directional properties of contours set a plurality of pixel regions with inputted R, B, G_a, and G_b pixels as each center to be evaluation regions, and performs the evaluation of the presence or absence of the contours for a pixel signal of the R pixel, B pixel, and G pixel in the evaluation regions and the evaluation of the directional properties of existing contours. Interpolation and high-frequency addition processing sections 16-19 calculate the interpolated value of two color elements, based on the values of a plurality of pixels in the two color elements that are at the center pixel, arranged in a direction indicated by determined directional properties and differ from the center pixel, when determining that the directional properties of a G signal and those of B/R signals are within a preset angle range, and interpolate the calculated interpolation values of the two color elements as pixel values of the two color elements in the center pixel. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, and more particularly to an image processing apparatus that performs pixel interpolation on Bayer array image data.

  2. Description of the Related Art A solid-state imaging device in which a Bayer array color filter is provided on a light receiving surface is known. As schematically shown in FIG. 21, this Bayer array color filter includes a red filter portion R for one pixel that transmits red light, a green filter portion G for one pixel that transmits green light, and blue light. The blue filter part B for 1 pixel which permeate | transmits is arrange | positioned. The Bayer array color filter shown in FIG. 21 has a checkered pattern of green filter portions G that transmit green light having a high contribution ratio of luminance signals in every other pixel pitch among the color filter portions R, G, and B described above. The red filter part R and the blue filter part B are arranged in a checkered pattern every other pixel pitch in the remaining portions.

  In this way, the color filter of the Bayer array is output from a solid-state imaging device in which the color filter of the Bayer array is provided on the light receiving surface because the color filter portion of the same color element exists at the geometrically skipped pixel position. The image processing apparatus that processes the captured image signal performs an interpolation process for interpolating the colors existing at the skipped pixel positions in order to obtain a signal in which the color elements of the three primary colors are aligned for all the pixels.

  Here, when the above-described image processing apparatus performs the above-described interpolation processing by calculating the average of simple neighboring pixels, jaggies and false colors (colors that do not exist in the original image) of the edge portion are widely generated. Are known. For this reason, various image processing apparatuses for reducing these jaggies and false colors have been proposed. Many of the proposals perform processing that controls the interpolation conditions based on the continuity and correlation of colors.

  For example, a technique that focuses on the correlation in an oblique direction has been conventionally proposed (see, for example, Patent Document 1 and Patent Document 2). Patent Document 1 discloses an image processing device that calculates gradients in a plurality of directions of pixel values and performs interpolation by changing the weights in the respective directions based on the gradients. Patent Document 2 discloses an image processing apparatus that performs interpolation by determining an interpolation coefficient based on the correlation with each value of the upper, lower, left, and right pixels of a pixel to be processed.

  In addition, a conventional image processing apparatus that performs countermeasure processing for a problem caused by a narrow band of a red signal R and a blue signal B among false color problems is known (see, for example, Patent Document 3). In the image processing apparatus described in Patent Document 3, by using the RGB correlation in the high frequency band, the high frequency band of the green signal G is added to the red signal R and the blue signal B, respectively, so that the false color is reduced. is doing.

JP-A-10-164602 Japanese Patent Laid-Open No. 11-177994 JP 2000-50290 A

  However, since these prior arts do not use empirical rules or general properties such as the high correlation between the primary colors of RGB, each of the primary colors of RGB is independently correlated with the correlation estimation and interpolation coefficients. In some cases, the suppression of jaggies and false colors at the edge portion is insufficient.

  In addition, the RGB correlation in the high frequency in Patent Document 3 does not always hold, and there are cases where there is no correlation and there is an inverse correlation. In these cases, simple addition processing adversely affects the image quality.

  The present invention has been made in view of the above points, and an object thereof is to provide an image processing apparatus that can perform pixel interpolation with less jaggy and smoothness with less false colors.

In order to achieve the above object, according to a first aspect of the present invention, a color element constituting an imaging signal output from a solid-state imaging device having a Bayer array color filter provided on a light receiving surface is one of the three primary colors. An image processing apparatus that performs image processing for interpolating primary color signals of two other color elements other than the known color elements based on values of surrounding pixels in each known pixel,
Based on the relationship between the color element of the input image signal and the color element of the surrounding pixel when that pixel is the central pixel, it is classified as one of multiple pixel types Pixel type classifying means, and for each pixel type classified by the pixel type classifying means, a preset pixel region consisting of a pixel of the input imaging signal and a plurality of peripheral pixels centered on the pixel The pixel type classification unit classifies the detection result by using a predetermined arithmetic expression using the value of each pixel in the pixel region, and detects the largest directionality of the contour component for all three primary colors. For each of the pixel types, the direction indicated by the direction of a predetermined primary color signal as a reference and the directionality of the remaining two primary color signals out of the directions indicated by the directionality detected by the detection means The determination means for determining whether or not the angle between the two directions is within a preset angle range, and when the determination means determines that the angle is within the angle range, the determined directionality is indicated Based on the values of a plurality of pixels of two color elements different from the central pixel in the center pixel arranged in the direction, the interpolation values of the two color elements are calculated, and the calculated interpolation values of the two color elements are calculated. Interpolating means for interpolating as pixel values of two color elements in the center pixel.

  In order to achieve the above object, in addition to the configuration of the first invention, the second invention includes the center pixel arranged in a direction indicated by the directionality determined to have an angle within the angle range. The high frequency components of the primary color signal of the same color element as the central pixel calculated based on the values of the plurality of pixels in the pixel area of the same color element as the pixel values of the two color elements Addition / subtraction means for adding / subtracting according to the sign of either positive or negative attached to the calculation result of the predetermined calculation formula used when obtaining the pixel value.

  Furthermore, in order to achieve the above-mentioned object, the third aspect of the invention relates to the interpolation means of the above-mentioned invention in the center pixel arranged in the direction indicated by the directionality determined by the determination means that the angle is within the angle range. The values of a plurality of pixels of two color elements different from the central pixel are weighted averaged for each of the two color elements, and the weighted average value is the maximum and minimum of the pixels of the color element around the central pixel. The interpolating value is limited by a value determined by the value. The present invention uses continuity and correlation in a broad sense, and uses the edge directionality of pixel values with emphasis on diagonal interpolation.

  According to the present invention, it is possible to perform pixel interpolation that is smooth and has few false colors with little jaggy making use of the correlation of pixels of the three primary colors. In addition, according to the present invention, it is possible to realize a wide band of a video signal after interpolation by considering forward correlation, non-correlation, and inverse correlation in pixel interpolation.

Next, an embodiment of the present invention will be described in detail with reference to the drawings.
FIG. 1 shows a block diagram of an embodiment of an image processing apparatus according to the present invention. An image processing apparatus 10 according to this embodiment is an image processing apparatus that processes an imaging signal output from a solid-state imaging device having a Bayer array color filter provided on a light receiving surface, and is configured to output a Bayer array output from the solid-state imaging device. Center pixel type classifying unit 11 that receives the image pickup signal as an input, presence / absence of a contour indicating whether or not a display image based on the image signal classified and output by the central pixel type classifying unit 11 includes a contour, Outline presence / orientation evaluation units 12, 13, 14, and 15 that independently evaluate the directionality indicating the direction in the display image of the contour, and the presence / absence / direction evaluation unit 12, 13, 14, and 15 of the contour, Interpolation / high-frequency addition processing units 16, 17, 18, and 19 that separately perform interpolation / high-frequency addition processing on the 15 output signals, and interpolation / high-frequency addition processing units 16, 17, 18, More composed achromatic processing unit 20, 21, 22, 23 for performing achromatic processing for each output signal of 9.

  The central pixel type classification unit 11 determines which pixel type is a known central pixel at the position of the color filter unit of the Bayer array image signal input from the solid-state imaging device, from among four types of pixel types described later. The center pixel is sorted and output according to the classified pixel type.

The presence / absence / direction evaluation unit 12, 13, 14, 15 of the contour evaluates the presence / absence of the contour of the input image signal, and further evaluates the directionality of the contour based on the evaluation result of the presence / absence of the contour.
The interpolation / high-frequency addition processing units 16, 17, 18, and 19 are provided in a one-to-one correspondence with the contour presence / absence / direction evaluation units 12, 13, 14, and 15. Based on the evaluation results of the directionality of the contours output from the sex evaluation units 12, 13, 14, and 15, interpolation processing of pixels of predetermined two color elements and high-frequency addition are performed.

  The achromatic color processing units 20, 21, 22, and 23 are provided in one-to-one correspondence with the interpolation / high-frequency addition processing units 16, 17, 18, and 19, and the interpolation / high-frequency addition processing unit. The processing results output from 16, 17, 18, and 19 are subjected to achromatic processing when a condition described later is satisfied, and output interpolation processing signals for pixels of predetermined two color elements, respectively.

  Next, the operation of the present embodiment will be described in detail with reference to the flowcharts of FIGS. 2A and 2B and FIGS. 3 to 20. The center pixel type classifying unit 11 receives as an input an imaging signal output from a solid-state imaging device having a Bayer array color filter provided on the light receiving surface (step S1 in FIG. 2A). Each pixel signal of the input image pickup signal is one of a red signal R, a green signal G, and a blue signal B based on the Bayer array shown in FIG. Further, the input image pickup signal is classified into four types shown in FIGS. 3A to 3D depending on which of R, B, and G the color element of the center pixel of the vertical direction 3 pixels and the horizontal direction 3 pixels is. be able to. Here, the center pixel is a known pixel corresponding to the color filter portion of the color filter in the Bayer array.

  FIG. 3A shows the case where the central pixel at the position of the color filter portion R shown in FIG. 21 is a pixel of the color element R of the three primary colors (hereinafter also referred to as R pixel). The pixels of the color element G of the three primary colors (hereinafter also referred to as G pixels), and the pixels in the left and right diagonal directions are the pixels of the color element B of the three primary colors (hereinafter also referred to as B pixels). FIG. 3B shows the case where the central pixel at the position of the color filter portion B shown in FIG. 21 is a B pixel, the upper and lower left and right pixels are G pixels, and the left and right diagonal pixels are R pixels. Pixel.

  When the known central pixel at the position of the color filter portion G shown in FIG. 21 is a G pixel, there are two types shown in FIGS. 3C and 3D depending on the upper, lower, left, and right pixels. The pixel arrays shown in FIGS. 3C and 3D are common in that the center pixel is a G pixel and the left and right diagonal pixels are also G pixels, but the pixel array in FIG. 3 is different from the pixel arrangement shown in FIG. 3D in that the upper and lower pixels are R pixels and the left and right pixels are B pixels. Here, the central pixel in the pixel array in FIG. 3C is referred to as a G_a pixel, and the central pixel in the pixel array in FIG. 3D is referred to as a G_b pixel.

  The central pixel type classifying unit 11 determines which type of the four types of pixels of the input image signal of the Bayer array is shown in FIGS. 3A to 3D, and the determined input pixel is R In the case of a pixel, an input pixel signal is supplied to the contour presence / absence / direction evaluation unit 12. Similarly, the center pixel type classification unit 11 determines whether or not there is a contour when the determined input pixel is a B pixel, and whether or not there is a contour when the determined input pixel is a G_a pixel. When the determined input pixel is a G_b pixel, the input pixel signal is supplied to the outline presence / orientation evaluation unit 15 (step S2 in FIG. 2A).

  The outline presence / absence / direction evaluation unit 12 uses the pixel range shown in FIGS. 4A to 4C of 7 pixels in the vertical direction and 7 pixels in the horizontal direction centered on the input R pixel as an evaluation region, and the evaluation is performed. Evaluation of the presence / absence of the outline of the pixel signal of the R pixel indicated by a bold circle in FIG. 4A of the region (step S3 in FIG. 2A) and the pixel signal of the B pixel indicated by the bold circle in FIG. The evaluation of the presence / absence of a contour (step S4 in FIG. 2A) and the evaluation of the presence / absence of a contour (step S5 in FIG. 2A) for the pixel signal of the G pixel indicated by a bold circle in FIG. 4C are performed. Subsequently, when the presence / absence / direction evaluation unit 12 of the outline evaluates that the outline exists in steps S3 to S5, the directionality of the existing outline is evaluated (steps S15 to S17 in FIG. 2A).

  Similarly, the outline presence / absence / direction evaluation unit 13 uses the pixel range shown in FIGS. 5A to 5C with 7 pixels in the vertical direction and 7 pixels in the horizontal direction centered on the input B pixel as an evaluation region. 5A, 5B, and 5C of the evaluation region is evaluated for the presence or absence of contours for the pixel signals of R, B, and G pixels indicated by bold circles (steps S6 to S6 in FIG. 2A). S8). Subsequently, when the presence / absence / direction evaluation unit 13 of the outline evaluates that the outline exists in steps S6 to S8, the directionality of the existing outline is evaluated (steps S18 to S20 in FIG. 2A).

  Further, the outline presence / absence / direction evaluation unit 14 uses the pixel range shown in FIGS. 6A to 6C of about 7 pixels in the vertical direction and 7 pixels in the horizontal direction centered on the input G_a pixel as an evaluation region. 6 (A), (B), and (C) of the evaluation region, the presence / absence of the outline of the pixel signals of the R pixel, B pixel, and G pixel indicated by bold circles is evaluated (steps S9 to S9 in FIG. 2A) S11). Subsequently, when the presence / absence / direction evaluation unit 15 of the outline evaluates that the outline exists in steps S9 to S11, the directionality of the existing outline is evaluated (steps S21 to S23 in FIG. 2A).

  Similarly, the presence / absence / direction evaluation unit 15 of the outline also uses the pixel range shown in FIGS. 7A to 7C of about 7 pixels in the vertical direction and 7 pixels in the horizontal direction around the input G_b pixel as the evaluation region. 7A, 7B, and 7C in the evaluation region, the presence / absence of an outline is evaluated for pixel signals of R pixels, B pixels, and G pixels indicated by bold circles (step S12 in FIG. 2A). To S14). Subsequently, when the presence / absence / direction evaluation unit 15 of the outline evaluates that the outline exists in steps S12 to S14, the directionality of the existing outline is evaluated (steps S24 to S26 in FIG. 2A).

  In effect, the presence / absence of the outline and the evaluation of the directionality are simultaneously performed by the outline presence / absence / direction evaluation units 12 to 15. The evaluation of the presence / absence of the outline and the evaluation of the directionality are performed for the pixels indicated by the bold circles shown in FIGS. 4 to 7, but the pixel arrangement differs depending on each pixel type and the color element to be evaluated. Therefore, multiple operator types must be prepared for the evaluation.

  Specifically, in the evaluation of the R signal in the R pixel, the operator type OP0 of 3 rows and 3 columns shown in FIG. 8 is used, and in the evaluation of the B signal in the R pixel, the 4 rows and 4 columns shown in FIG. The operator type OP1 shown in FIG. 10 is used in the evaluation of the G signal in the R pixel using the operator type OP1. In the evaluation of the R signal in the B pixel, the operator type OP1 shown in FIG. 9 is used. In the evaluation of the B signal in the B pixel, the liquidator type OP0 shown in FIG. 8 is used, and the evaluation of the G signal in the B pixel is performed. In FIG. 10, an operator type OP2 shown in FIG. 10 is used.

  Further, in the evaluation of the R signal in the G_a pixel, the operator type OP3 of 3 rows and 4 columns shown in FIG. 11 is used, and in the detail of the B signal in the G_a pixel, the operator type 0P4 of 4 rows and 3 columns shown in FIG. In the evaluation of the G signal in the G_a pixel, the operator type OP5 shown in FIG. 13 is used. Furthermore, in the evaluation of the R signal in the G_b pixel, the operator type OP4 shown in FIG. 12 is used, and in the evaluation of the B signal in the G_b pixel, the operator type OP3 shown in FIG. 11 is used, and in the evaluation of the G signal in the G_b pixel. Uses the operator type OP5 shown in FIG.

  For each of the operator types OP0 to OP5, eight edge directions of 0 °, 22.5 °, 45 °, 67.5 °, 90 °, 112.5 °, 135 °, and 157.5 ° are provided. Construct a differential operator that evaluates sex. 14 to 19 show eight-way differential operator groups for the operator types OP0 to OP5. In FIGS. 14 to 19, a circle indicates a pixel as a differential operator, and a number on the right shoulder of each pixel indicates a pixel number. Moreover, the straight lines shown in FIGS. 14A to 19B are respectively 0 °, 0 °, and (C), (D), (E), (F), (G), and (H). The directivity of 22.5 °, 45 °, 67.5 °, 90 °, 112.5 °, 135 °, 157.5 ° is shown.

  Here, the pixel value of pixel number i in FIGS. 14 to 19 is X [i], and the operator coefficient c [i] is c [i] = 1 for white circle pixels and c [i] for black circle pixels. = −1, c [i] = 0 for the hatched pixels, No as the total number of pixels of the operator, Nw as the number of white circle pixels, and Nb as the number of black circle pixels, and the correlation in the operator is ( 1) Evaluation is performed in a form normalized by the equation.

  In the evaluation of directionality, the calculation by the calculation formula (1) is performed for each of the above eight angles, and the angle indicating the maximum value is evaluated as having the directionality of the angle. The value of equation (1) represents the relative inclination of the pixel value in a certain direction under the condition of using the same operator.

  If the calculation result of the expression (1) does not exceed a predetermined threshold at any angle, it is determined that there is no contour. Above, the process of said step S15-S17, S18-S20, S21-S23, S24-S26 is complete | finished.

  Subsequently, the contour presence / absence / direction evaluation units 12 to 15 determine whether they are close by comparing two directions by color within the same pixel type (steps S27 to S28 and S29 to FIG. 2A). S30, S31 to S32, S34 to S35). Here, “the directionality is close” means that the two directions to be compared are the same or within ± 22.5 °.

  That is, the contour presence / orientation evaluation unit 12 for the pixel type R pixel determines whether or not the directionality of the R signal and the directionality of the G signal are close (step S27), and the directionality of the B signal. It is determined whether or not the directivity of the G signal is close (step S28). Further, the outline presence / orientation evaluation unit 13 for the pixel type B pixel determines whether or not the directionality of the R signal and the directionality of the G signal are close (step S29), and the directionality of the B signal. It is determined whether or not the directivity of the G signal is close (step S30). Further, the outline presence / orientation evaluation unit 14 for the pixel type G_a pixel determines whether or not the directionality of the R signal and the directionality of the G signal are close (step S31), and the directionality of the B signal. It is determined whether or not the directivity of the G signal is close (step S32). Further, the contour presence / orientation evaluation unit 15 for the pixel type G_b pixel determines whether or not the directionality of the R signal and the directionality of the G signal are close (step S34), and the directionality of the B signal. It is determined whether or not the directivity of the G signal is close (step S35).

  Based on the information on the presence / absence of the contour and the result of the directionality comparison, the determination in the subsequent processes is performed. Here, hereinafter, when “X is interpolated with the direction of Y”, the above-mentioned directionality calculated using an operator suitable for Y with respect to Y is the following of an operator type suitable for X with respect to X: This means that the result of the operation expressed in the C language is used as an interpolation value.

  Here, for each of the operator types OP1 to OP4, the interpolation value of the unknown color element of the central pixel in the eight directions is a weighted average of the values of the pixels along the direction as follows: Further, the weighted average value is represented by a limit value limited by a value determined by the maximum value and the minimum value of a plurality of peripheral pixels of the central pixel.

  In the following description, p [i] indicates a pixel number i, which is the same as that in FIGS. Max (,,,) indicates the maximum value in parentheses, and min (,,,) indicates the minimum value in parentheses. Further, limit_min_max (X, min1, max1) means that X is limited with min1 as the lower limit and max1 as the upper limit. Further, the interpolation value is “intp”.

(1) About OP1
max1 = max (p [5], p [6], p [9], p [10]); min1 = min (p [5], p [6], p [9], p [10]); thr = max1-min1;
// OP1-0 °: Simple average of the nearest 4 pixels
intp = (p [5] + p [6] + p [9] + p [10] +2) / 4;
//OP1-22.5°:
if (abs (p [6] -p [7]) <thr && abs (p [8] -p [9]) <thr)
intp = limit_min_max ((p [6] + p [7] + p [8] + p [9] +2) / 4, min1, max1);
else
intp = (p [6] + p [9] +1) / 2;
// OP1-45 °: Simple average of two diagonal pixels
intp = (p [6] + p [9] +1) / 2;
//OP1-67.5°:
if (abs (p [2] -p [6]) <thr && abs (p [9] -p [13]) <thr)
intp = limit_min_max ((p [2] + p [6] + p [9] + p [13] +2) / 4, min1, max1);
else
intp = (p [6] + p [9] +1) / 2;
// OP1-90 °: Simple average of the nearest 4 pixels
intp = (p [5] + p [6] + p [9] + p [10] +2) / 4;
//OP1-112.5°:
if (abs (p [1] -p [5]) <thr && abs (p [10] -p [14]) <thr)
intp = limit_min_max ((p [1] + p [5] + p [10] + p [14] +2) / 4, min1, max1);
else
intp = (p [5] + p [10] +1) / 2;
// OP1-135 °: Simple average of two diagonal pixels
intp = (p [5] + p [10] +1) / 2;
//OP1-157.5°:
if (abs (p [4] -p [5]) <thr && abs (p [10] -p [11]) <thr)
intp = limit_min_max ((p [4] + p [5] + p [10] + p [11] +2) / 4, min1, max1);
else
intp = (p [5] + p [10] +1) / 2;
(2) About OP2
max1 = max (p [8], p [11], p [12], p [15]); min1 = min (p [8], p [11], p [12], p [15]);
thr = max1-min1;
// OP2-0 °: Simple average of left and right 2 pixels
intp = (p [11] + p [12] +1) / 2;
//OP2-22.5°:
if (abs (p [9] -p [12]) <thr && abs (p [11] -p [14]) <thr)
intp = limit_min_max ((p [9] + p [11] + p [12] + p [14] +2) / 4, min1, max1);
else
intp = (p [11] + p [12] +1) / 2;
// OP2-45 °: Simple average of the nearest 4 pixels
intp = (p [8] + p [11] + p [12] + p [15] +2) / 4;
//OP2-67.5°:
if (abs (p [5] -p [8]) <thr && abs (p [15] -p [18]) <thr)
intp = limit_min_max ((p [5] + p [8] + p [15] + p [18] +2) / 4, min1, max1);
else
intp = (p [8] + p [15] +1) / 2;
// OP2-90 °: Simple average of upper and lower two pixels
intp = (p [8] + p [15] +1) / 2;
//OP2-112.5°:
if (abs (p [4] -p [8]) <thr && abs (p [15] -p [19]) <thr)
intp = limit_min_max ((p [4] + p [8] + p [15] + p [19] +2) / 4, min1, max1);
else
intp = (p [8] + p [15] +1) / 2;
// OP2-135 °: Simple average of the nearest 4 pixels
intp = (p [8] + p [11] + p [12] + p [15] +2) / 4;
//OP2-157.5°:
if (abs (p [7] -p [11]) <thr && abs (p [12] -p [16]) <thr)
intp = limit_min_max ((p [7] + p [11] + p [12] + p [16] +2) / 4, min1, max1);
else
intp = (p [11] + p [12] +1) / 2;
(3) About OP3
max1 = max (p [5], p [6]); min1 = min (p [5], p [6]); thr = max1-min1;
// OP3-0 °: Simple average of left and right 2 pixels
intp = (p [5] + p [6] +1) / 2;
//OP3-22.5°:
if (abs (p [3] -p [6]) <thr && abs (p [5] -p [8]) <thr)
intp = limit_min_max ((p [3] + p [6] + p [5] + p [8] +2) / 4, min1, max1);
else
intp = (p [5] + p [6] +1) / 2;
// OP3-45 °: Simple average of the nearest 4 pixels
if (abs (p [2] -p [6]) <thr && abs (p [5] -p [9]) <thr)
intp = limit_min_max ((p [2] + p [6] + p [5] + p [9] +2) / 4, min1, max1);
else
intp = (p [5] + p [6] +1) / 2;
//OP3-67.5°:
intp = limit_min_max ((p [2] + p [9] +1) / 2, min1, max1);
// OP3-90 °: Simple average of left and right 2 pixels
intp = (p [5] + p [6] +1) / 2;
//OP3-112.5°:
intp = limit_min_max ((p [1] + p [10] +1) / 2, min1, max1);
// OP3-135 °: Simple average of the nearest 4 pixels
if (abs (p [1] -p [5]) <thr && abs (p [6] -p [10]) <thr)
intp = limit_min_max ((p [1] + p [5] + p [6] + p [10] +2) / 4, min1, max1);
else
intp = (p [5] + p [6] +1) / 2;
//OP3-157.5°:
if (abs (p [0] -p [5]) <thr && abs (p [6] -p [11]) <thr)
intp = limit_min_max ((p [0] + p [5] + p [6] + p [11] +2) / 4, min1, max1);
else
intp = (p [5] + p [6] +1) / 2;
(4) About OP4
max1 = max (p [4], p [7]); min1 = min (p [4], p [7]); thr = max1-min1;
// OP4-0 °: Simple average of upper and lower two pixels
intp = (p [4] + p [7] +1) / 2;
//OP4-22.5°:
intp = limit_min_max ((p [5] + p [6] +1) / 2, min1, max1);
// OP4-45 °:
if (abs (p [4] -p [5]) <thr && abs (p [6] -p [7]) <thr)
intp = limit_min_max ((p [4] + p [5] + p [6] + p [7] +2) / 4, min1, max1);
else
intp = (p [4] + p [7] +1) / 2;
//OP4-67.5°:
if (abs (p [2] -p [4]) <thr && abs (p [7] -p [9]) <thr)
intp = limit_min_max ((p [2] + p [4] + p [7] + p [9] +2) / 4, min1, max1);
else
intp = (p [4] + p [7] +1) / 2;
// OP4-90 °: Simple average of upper and lower two pixels
intp = (p [4] + p [7] +1) / 2;
//OP4-112.5°:
if (abs (p [0] -p [4]) <thr && abs (p [7] -p [11]) <thr)
intp = limit_min_max ((p [0] + p [4] + p [7] + p [11] +2) / 4, min1, max1);
else
intp = (p [4] + p [7] +1) / 2;
// OP4-135 °:
if (abs (p [3] -p [4]) <thr && abs (p [7] -p [8]) <thr)
intp = limit_min_max ((p [3] + p [4] + p [7] + p [8] +2) / 4, min1, max1);
else
intp = (p [4] + p [7] +1) / 2;
//OP4-157.5°:
intp = limit_min_max ((p [3] + p [8] +1) / 2, min1, max1);
Here, for the operator types OP1 and OP2, the pixel values of the nearest four pixels of the interpolated pixel (center pixel), and for the operator types OP3 and OP4, the pixel values of the nearest two pixels of the interpolated pixel (center pixel). The maximum value and the minimum value are limited by this, but this is to prevent the interpolated value from protruding. However, the present invention is not limited to using the maximum value / minimum value of the pixel value of the nearest pixel, and other limit values can be used.

  Also, OP1-22.5 °, OP1-67.5 °, OP1-1122.5 °, OP1-1-157.5 °, OP2-22.5 °, OP2-67.5 °, OP2-12.5 ° , OP2-157.5 °, OP3-22.5 °, OP3-45 °, OP3-135 °, OP3-157.5 °, OP4-45 °, OP4-67.5 °, OP4-1122.5 ° As for the pair of pixel values used for the four-value weighted average at OP4-135 ° on both sides in the contour direction, if the difference exceeds the difference between the maximum value and the minimum value, there is a large risk that the interpolation value may be disturbed by the protruding value. Switch to interpixel average interpolation. This is not limited to using the maximum and minimum pixel values of the nearest pixels, and other limit values can be used. Note that the operator types OP0 and OP5 already have the pixel value of the center pixel already known, and therefore the process for obtaining the interpolated value of the unknown color element in the center pixel by the weighted average calculation as described above is unnecessary.

  Returning to the description of the determination process based on the information on the presence / absence of the contour and the comparison result of the directionality. The interpolation / high-frequency addition processing units 16 to 19 in FIG. 1 determine whether or not the condition for the case where the directionality of the R signal or B signal is not significantly inconsistent with the directionality of the G signal (step S37 in FIG. 2B). , S48, S59, S66), an interpolation process and a high-frequency addition process are performed according to the determination result (steps S38 to S45, S49 to S56, S60 to S63, and S67 to S70 in FIG. 2B).

  For example, when the interpolation / high-frequency addition processing unit 16 for the pixel type R pixel determines that the condition of the direction of the R signal or the B signal is not significantly inconsistent with the direction of the G signal (step S37). Y), the directionality of the G signal is preferentially adopted, and the G signal is first interpolated with the directionality of the G signal (step S38). Here, the condition in step S37 is that the contour of the R signal is not present, the contour of the G signal and the B signal are present and close to each other, or the contour of the B signal is absent, and the G signal and the R signal are present. When there is a contour of the signal and the directionality is close, or when there are contours of the R signal, the G signal and the B signal, and the directionality of the R signal and the G signal and the directionality of the B signal and the G signal are close, or R This is a case where there is no outline of the signal and B signal and there is an outline of the G signal.

  Subsequently, the interpolation / high-frequency addition processing unit 16 similarly interpolates the BG signal with the directionality of the G signal (step S39). The G signal is interpolated using the operator type OP2, and the B-G signal is interpolated using the operator type OP1 for the value obtained by subtracting the corresponding G result from the B value of the neighboring pixel. Subsequently, the interpolation / high-frequency addition processing unit 16 adds the high-frequency component of the R signal to the interpolation result of the G signal (step S40). This will be described later. Similarly, the interpolation / high-frequency addition processing unit 16 adds the high-frequency component of the G signal to the B signal obtained as the sum of the interpolation result of the B-G signal and the interpolation result of the G signal (step) S41).

  On the other hand, when none of the four determination conditions in step S37 is satisfied (N in step S37), the interpolation / high-frequency addition processing unit 16 greatly contradicts the directionality of the R signal or B signal with the directionality of the G signal. Then, the G signal and the BG signal are interpolated with the average value of the pixel values of the nearest pixels of the R pixel (steps S42 and S43). Subsequently, the interpolation / high-frequency addition processing unit 16 adds the high-frequency component of the R signal to the interpolation result of the G signal (step S44), and the interpolation result of the B-G signal and the G signal The high frequency component of the G signal is added to the B signal obtained as the sum of the interpolation results (step S45). Then, the interpolation / high-frequency addition processing unit 16 outputs the processing result of step S41 or S45 to the achromatic processing unit 20 for a processing for achromatizing a false color to be described later, and performs the achromatic processing. (Step S46 in FIG. 2B).

  Further, when the interpolation / high-frequency addition processing unit 17 for the pixel type B pixel determines that the condition of the direction of the R signal or the B signal is not significantly inconsistent with the direction of the G signal (step S48). Y), the directionality of the G signal is preferentially adopted, and the G signal is first interpolated with the directionality of the G signal (step S49). Here, the condition in step S48 is that the contour of the R signal is not present, the contour of the G signal and the B signal are present, and the directionality is close, or the contour of the B signal is not present, and the G signal and R When there is a contour of the signal and the directionality is close, or when there are contours of the R signal, the G signal and the B signal, and the directionality of the R signal and the G signal and the directionality of the B signal and the G signal are close, or R This is a case where there is no outline of the signal and B signal and there is an outline of the G signal.

  Subsequently, the interpolation / high-frequency addition processing unit 17 similarly interpolates the RG signal with the directionality of the G signal (step S50). The G signal is interpolated using the operator type OP2, and the RG signal is interpolated using the operator type OP1 for a value obtained by subtracting the corresponding G result from the R value of the neighboring pixel. Subsequently, the interpolation / high-frequency addition processing unit 17 adds the high-frequency component of the B signal to the interpolation result of the G signal (step S51). This will be described later. Similarly, the interpolation / high-frequency addition processing unit 17 adds the high-frequency component of the G signal to the R signal obtained as the sum of the interpolation result of the RG signal and the interpolation result of the G signal (Step S1). S52).

  On the other hand, when none of the four determination conditions in step S48 is satisfied (N in step S48), the direction of the R signal or the B signal is largely inconsistent with the direction of the G signal. Then, the G signal and the RG signal are interpolated with the pixel average value of the nearest pixel of the B pixel (steps S53 and S54). Subsequently, the interpolation / high-frequency addition processing unit 17 adds the high-frequency component of the B signal to the interpolation result of the G signal (step S55), and the interpolation result of the RG signal and the G signal The high frequency component of the G signal is added to the R signal obtained as the sum of the interpolation results (step S56). Then, the interpolation / high-frequency addition processing unit 17 outputs the processing result of step S52 or S56 to the achromatic processing unit 21 for achromatic processing to be described later, and achromatic processing is performed. (Step S57 in FIG. 2B).

  Further, when the interpolation / high-frequency addition processing unit 18 for the pixel type G_a pixel determines that the condition of the direction of the R signal or the B signal is not significantly inconsistent with the direction of the G signal (step S59). Y), the directionality of the G signal is preferentially adopted, and both the RG signal and the BG signal are interpolated with the directionality of the G signal (step S60). Here, the condition in step S59 described above is that the contour of the R signal is not present, the contour of the G signal and the B signal are present, and the directionality is close, or the contour of the B signal is absent, and the G signal and R When there is a contour of the signal and the directionality is close, or when there are contours of the R signal, the G signal and the B signal, and the directionality of the R signal and the G signal and the directionality of the B signal and the G signal are close, or R This is a case where there is no outline of the signal and B signal and there is an outline of the G signal. Note that the interpolation uses the operator type OP3 for the RG signal obtained by subtracting the corresponding G result from the R value of the neighboring pixel, and for the BG signal, the B of the neighboring pixel. The value obtained by subtracting the corresponding G result from the value is calculated using the operator type OP4.

  Subsequently, the interpolation / high-frequency addition processing unit 18 adds the high-frequency component of the G signal to the R signal obtained as the sum of the interpolation result of the RG signal and the interpolation result of the G signal, The high frequency component of the G signal is added to the B signal obtained as the sum of the interpolation result of the BG signal and the interpolation result of the G signal (step S61). This will be described later.

  On the other hand, when none of the four determination conditions in step S59 is satisfied (N in step S59), the interpolation / high-frequency addition processing unit 18 greatly contradicts the directionality of the R signal or B signal with the directionality of the G signal. For the RG signal, interpolation is performed with the average value of the right and left R pixels of the G pixel as the central pixel, and for the BG signal, the average value of the B pixels above and below the G pixel as the central pixel. Is interpolated (step S62). Subsequently, the interpolation / high-frequency addition processing unit 18 adds the high-frequency component of the G signal to the R signal and B signal that are the interpolation results (step S63). Then, the interpolation / high-frequency addition processing unit 18 outputs the processing result of step S61 or S63 to the achromatic processing unit 22 for processing to achromaticize the false color described later, and performs the achromatic processing. (Step S64 in FIG. 2B).

  Further, when the interpolation / high-frequency addition processing unit 19 for the pixel type G_b pixel determines that the condition of the direction of the R signal or the B signal is not significantly inconsistent with the direction of the G signal (step S66). Y), the directionality of the G signal is preferentially adopted, and both the RG signal and the BG signal are interpolated by the directionality of the G signal (step S67). Here, the condition in step S66 is that the contour of the R signal is not present, the contour of the G signal and the B signal are present, and the directionality is close, or the contour of the B signal is absent, and the G signal and R When there is a contour of the signal and the directionality is close, or when there are contours of the R signal, the G signal and the B signal, and the directionality of the R signal and the G signal and the directionality of the B signal and the G signal are close, or R This is a case where there is no outline of the signal and B signal and there is an outline of the G signal. Note that the interpolation uses the operator type OP4 for the RG signal obtained by subtracting the corresponding G result from the R value of the neighboring pixel, and for the BG signal, the B of the neighboring pixel. The value obtained by subtracting the corresponding G result from the value is calculated using the operator type OP3.

  Subsequently, the interpolation / high-frequency addition processing unit 19 adds the high-frequency component of the G signal to the R signal obtained as the sum of the interpolation result of the RG signal and the interpolation result of the G signal, The high frequency component of the G signal is added to the B signal obtained as the sum of the interpolation result of the B-G signal and the interpolation result of the G signal (step S68). This will be described later.

  On the other hand, when none of the four determination conditions in step S66 is satisfied (N in step S66), the interpolation / high-frequency addition processing unit 19 greatly contradicts the directionality of the R signal or B signal with the directionality of the G signal. For the RG signal, interpolation is performed with the average value of the upper and lower R pixels of the G pixel as the central pixel, and for the BG signal, the average value of the left and right B pixels of the G pixel as the central pixel. Is interpolated (step S69). Subsequently, the interpolation / high-frequency addition processing unit 19 adds the high-frequency component of the G signal to the R signal and B signal that are the interpolation results (step S70). Then, the interpolation / high-frequency addition processing unit 19 outputs the processing result of step S68 or S70 to the achromatic processing unit 23 for achromatic processing to be described later, and achromatic processing is performed. (Step S71 in FIG. 2B).

  Next, the high-frequency addition processing in steps S40, S41, S44, S45, S51, S52, S55, S56, S61, S63, S68, and S70 will be described.

  When the contour directionality of the unknown color matches the known color (when there is a correlation), it is determined that the high-frequency correlation of each is high, and the already obtained interpolation value (P [j] [i]) is used. ) Is added with a high frequency component as follows. Here, the following formula (2) is defined by removing the absolute value symbol of the numerator of formula (1). The correlation between the forward correlation and the inverse correlation can be determined by the sign of the equation (2).

(A) In the case of R pixel and B pixel Operator type OP0 used for evaluation of R signal in R pixel and evaluation of B signal in B pixel, evaluation of G signal in R pixel, and evaluation of G signal in B pixel When the signs of the calculation results of the expression (2) for the operator type OP2 used in the above are the same, a forward correlation is obtained in which the contours match and the pixel values increase and decrease in the direction perpendicular to the contours match. If the codes do not match, an inverse correlation is assumed. In the case of forward correlation, the high frequency component of the R / B signal of the R / B pixel of the known color element is added to the unknown G signal. In the case of inverse correlation, the high frequency component of the R / B signal of the R / B pixel of the known color element is subtracted from the unknown G signal (the high frequency component of the reverse phase of the R / B signal is added) (step S40, S44, S51, S55).

  In deriving the high frequency component hp, the pixel value of the pixel number i in the differential operator type group of OP0 of FIG. 14 is calculated as p [i] according to the directionality of interpolation as follows.

0 °: hp = (2 * p [4]-p [3]-p [5] +1) / 2;
22.5 °: hp = (2 * p [4]-p [2]-p [6] +1) / 2;
45 °: hp = (2 * p [4]-p [2]-p [6] +1) / 2;
67.5 °: hp = (2 * p [4]-p [2]-p [6] +1) / 2;
90 °: hp = (2 * p [4]-p [1]-p [7] +1) / 2;
112.5 °: hp = (2 * p [4]-p [0]-p [8] +1) / 2;
135 °: hp = (2 * p [4]-p [0]-p [8] +1) / 2;
157.5 °: hp = (2 * p [4]-p [0]-p [8] +1) / 2;
The same high-frequency extraction is performed on the G signal obtained as described above and added to the B / R signal (steps S41, S45, S52, and S56).

(B) In the case of G_a pixel and G_b pixel Operator type OP3 used for evaluation of R signal in G_a pixel and evaluation of B signal in G_b pixel, evaluation of B signal in G_a pixel, and evaluation of R signal in G_b pixel When the sign of the calculation result of the expression (2) is the same for the operator type OP4 used for the calculation of the G signal in the G_a pixel and the operator type OP5 used for the G signal evaluation in the G_b pixel, the contour is A forward correlation in which the pixel values increase and decrease in the direction perpendicular to the contour match, and a negative correlation is used if the signs do not match. In the case of forward correlation, the high frequency component of the G signal of the G pixel of the known color element is added to the R / B signal of the unknown color. In the case of inverse correlation, the high-frequency component of the G signal of the G pixel of the known color element is subtracted from the R / B signal of the unknown color (the high-frequency component of the opposite phase of the G signal is added) (steps S61, S63, S68, S70).

  In deriving the high-frequency component hp, the pixel value of the pixel number i in the differential operator type group of OP5 in FIG. 19 is calculated as p [i] according to the directionality of interpolation as follows.

0 °: hp = (2 * p [10]-p [9]-p [11] +1) / 2;
22.5 °: hp = (2 * p [10]-p [7]-p [13] +1) / 2;
45 °: hp = (2 * p [10]-p [7]-p [13] +1) / 2;
67.5 °: hp = (2 * p [10]-p [7]-p [13] +1) / 2;
90 °: hp = (2 * p [10]-p [3]-p [17] +1) / 2;
112.5 °: hp = (2 * p [10]-p [6]-p [14] +1) / 2;
135 °: hp = (2 * p [10]-p [6]-p [14] +1) / 2;
157.5 °: hp = (2 * p [10]-p [6]-p [14] +1) / 2;
Further, when interpolation based on directionality is not performed, interpolation is performed by nearest-neighbor linear interpolation, and therefore, a high frequency component is acquired from the values of neighboring pixels using a linear filter.

  Next, the false color achromatic color processing in steps S46, S57, S64, and S71 by the achromatic color processing units 20 to 23 will be described.

  The G signal includes a horizontal and vertical stripe-shaped high-frequency variation pattern due to the Bayer array color filter, which causes generation of false colors. Perform a neutralization process.

Specifically, as shown in FIGS. 20A and 20B, the achromatic color processing units 20 and 21 perform horizontal processing for G pixels in the operator type OP2 when the central pixel is an R pixel or a B pixel. Average values of pixel values for three pixels or four pixels are calculated in the vertical direction and the vertical direction, respectively. Further, as shown in FIGS. 20C and 20D, the achromatic color processing units 22 and 23 are arranged in the horizontal and vertical directions for G pixels in the operator type OP5 when the central pixel is a G_a pixel or a G_b pixel. Average values of pixel values for three pixels or four pixels are calculated in the direction. Then, the achromatic color processing units 20 to 23 determine that the average value variation exceeds a certain value and determine that the pattern is a striped high-frequency variation pattern, R pixels, G pixels, and B pixels that have already been determined. Based on the pixel values R, G, and B of the pixel, the following equation Y = 0.715G + 0.0772B + 0.2126R (3)
The signal Y is generated by the above, and this is used as the pixel value of the new R pixel, G pixel, and B pixel, thereby achromatic.

  In this way, the achromatic color processing unit 20 outputs the G signal and the B signal obtained by the achromatic color processing (step S47 in FIG. 2B). Further, the achromatic color processing unit 21 outputs the G signal and the R signal obtained by the achromatic color processing (step S58 in FIG. 2B). Further, the achromatic color processing units 22 and 23 respectively output the R signal and the B signal obtained by the achromatic color processing (Steps S65 and S72 in FIG. 2B).

  As described above, according to the present embodiment, the imaging signals output from the solid-state imaging device in which the Bayer array color filter is provided on the light receiving surface are classified into four types of pixel types, and the classified pixel types are classified. The directionality of the contour of each signal is detected based on the values of the R pixel, G pixel, B pixel and surrounding pixels in the image pickup signal, and the directionality of the R signal or B signal for the detected directionality If there is no significant contradiction with the directionality of the G signal, the directionality of the G signal is preferentially adopted, and further used for pixel value interpolation of the R signal or B signal, so that the R signal, G signal, B It is possible to perform pixel interpolation with little jaggy and little false color utilizing the correlation of signals.

  Further, according to the present embodiment, when the weighted average of pixel values along the directionality detected by the presence / absence / directionality evaluation units 12 to 15 of the contour is used as the interpolation value, the weighted average value is set to a predetermined number. By limiting by the value determined by the maximum value and the minimum value of the surrounding pixels, it is possible to avoid protruding interpolation values.

  Furthermore, according to the present embodiment, when the signs of the calculation results of equation (2) are the same, the contours match and the pixel values increase and decrease in the direction perpendicular to the contours match. Correlation is assumed, and if the codes do not match, the inverse correlation is assumed. And the high frequency component of the primary color signal of the same color element as the center pixel, calculated based on the values of a plurality of pixels in the pixel area of the same color element as the center pixel arranged in the direction indicated by the directionality, In the case of forward correlation, the pixel values of the B pixel and R pixel are added when obtaining these pixel values, and in the case of inverse correlation, they are subtracted. By doing so, it is possible to realize a wide band of the video signal after interpolation.

  The present invention is not limited to an image processing apparatus that performs hardware Bayer image interpolation processing, and includes a computer program that causes a computer to execute the configuration of the block diagram of FIG. The image processing program which is a computer program in this case may be taken into a computer from a recording medium, or may be taken into a computer via a network.

It is a block diagram of one embodiment of an image processing device of the present invention. FIG. 3 is a flowchart (part 1) for explaining the operation of FIG. 1; FIG. FIG. 3 is a flowchart (part 2) for explaining the operation of FIG. 1; It is a figure explaining classification of a pixel type in the present invention. It is a figure which shows RGB pixel arrangement | positioning of R pixel vicinity. It is a figure which shows RGB pixel arrangement | positioning of B pixel vicinity. It is a figure which shows RGB pixel arrangement | positioning of G_a pixel vicinity. It is a figure which shows the RGB pixel arrangement | positioning of G_b pixel vicinity. It is a figure which shows pixel arrangement | positioning of operator type OP0. It is a figure which shows pixel arrangement | positioning of operator type OP1. It is a figure which shows pixel arrangement | positioning of operator type OP2. It is a figure which shows pixel arrangement | positioning of operator type OP3. It is a figure which shows pixel arrangement | positioning of operator type OP4. It is a figure which shows pixel arrangement | positioning of operator type OP5. It is a figure which shows the differential operator type group using the pixel arrangement | positioning of operator type OP0. It is a figure which shows the differential operator type group using the pixel arrangement | positioning of operator type OP1. It is a figure which shows the differential operator type group using the pixel arrangement | positioning of operator type OP2. It is a figure which shows the differential operator type group using the pixel arrangement | positioning of operator type OP3. It is a figure which shows the differential operator type group using the pixel arrangement | positioning of operator type OP4. It is a figure which shows the differential operator type group using the pixel arrangement | positioning of operator type OP5. It is a figure for demonstrating the detection method of the area | region which should be achromatic for the false color removal. It is a figure which shows an example of a Bayer arrangement | sequence.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Image processing apparatus 11 Center pixel type classification | category part 12-15 Presence / absence / directivity evaluation part 16-19 Interpolation / high-frequency addition processing part 20-23 Achromatic processing part

Claims (3)

In each known pixel in which the color element constituting the imaging signal output from the solid-state imaging device having the Bayer array color filter provided on the light receiving surface is one of the three primary colors, other than the known color element An image processing apparatus for performing image processing for interpolating primary color signals of other two color elements based on the values of surrounding pixels,
Based on the relationship between the color element of the input image signal and the color element of the surrounding pixel when the pixel is the central pixel, which of the plurality of types of pixel types is determined Pixel type classification means for classifying;
For each pixel type classified by the pixel type classifying means, the pixel within a preset pixel area consisting of a pixel of the input imaging signal and a plurality of peripheral pixels centered on the pixel. Detection means for calculating the maximum directionality of the contour component for all the primary color signals, using a value of each pixel in the region and calculating with a predetermined calculation formula;
For each of the pixel types classified by the pixel type classification unit, among the directions indicated by the directionality detected by the detection unit, the direction indicated by the direction of a predetermined primary color signal as a reference and the remaining two Determination means for respectively determining whether or not the angle between the directions indicated by the directions of the two primary color signals is within a preset angle range;
When the determination unit determines that the angle is within the angle range, a plurality of pixels having two color elements different from the central pixel in the central pixel arranged in the direction indicated by the determined directionality. Interpolation means for calculating an interpolation value of the two color elements based on the value, and interpolating the calculated interpolation values of the two color elements as pixel values of the two color elements in the central pixel;
An image processing apparatus comprising:   The center pixel calculated based on the values of a plurality of pixels in the pixel area of the same color element as the center pixel arranged in the direction indicated by the directionality determined that the angle is within the angle range The high frequency component of the primary color signal of the same color element as the pixel value of the two color elements is either positive or negative attached to the calculation result of the predetermined arithmetic expression used when obtaining these pixel values 2. The image processing apparatus according to claim 1, further comprising addition / subtraction means for adding / subtracting according to the sign. The interpolating means includes
The values of a plurality of pixels of two color elements different from the central pixel in the central pixel arranged in the direction indicated by the directionality determined by the determination means that the angle is within the angular range are calculated by the two values. Means for calculating a weighted average for each color element, and further using the weighted average value as a value obtained by limiting the weighted average value by a value determined by a maximum value and a minimum value of a plurality of pixels of the color element around the center pixel. The image processing apparatus according to claim 1, wherein the image processing apparatus is provided.

Images