JP6299104B2 - Image processing apparatus, camera, and image processing program - Google Patents

Description

  The present invention relates to an image processing apparatus, a camera, and an image processing program.

  When generating an image by performing color interpolation processing based on a raw image taken by an image sensor equipped with a color filter array such as a Bayer array, when false colors such as color moire occur in a structure with high-frequency components There is. For this reason, conventionally, processing for removing false colors by color difference smoothing has been performed.

JP 2009-200355 A

  When color difference smoothing is performed, the saturation of a subject with color may decrease or color blur may occur around the subject. In order to remove color moiré, color difference smoothing that refers to a wide range is required, but the above-mentioned problem becomes prominent and there is a problem that the reference range for color difference smoothing cannot be widened.

An image processing apparatus according to a first aspect of the invention includes a first color pixel having a spectral sensitivity of a first color component and a second color component having a spectral sensitivity of a second color component different from the first color component. An image input unit that acquires image data output from an image sensor in which color pixels are arranged, and a first color difference of the first color pixel in a first range including the first color pixel A first color difference generation unit that generates by performing smoothing, and a second color difference that the first color pixel has is a second color difference that is smaller than the first range and includes the first color pixel. A second color difference generation unit that generates based on the second color component of the second color pixel located in the range, and the first color difference value is in a distribution range of the second color difference value. When included, the first color difference is included, and when the distribution range does not include the first color difference value A color difference smoothing control unit that generates a boundary value of the distribution range as a third color difference of the first color pixel, the second color component of the second color pixel, and the first color An image generation unit that generates an output image based on the third color difference of the pixel, and the second color difference generation unit is a target for generating the second color difference. The first color component included in the first color pixel and the second color pixel included in the second range and positioned above and below the first color pixel respectively. A color difference generation process that is generated based on a color component, the first color component that the first color pixel that is the target for generating the second color difference has the second color difference, and the second color difference. And the second color pixels located to the left and right of the first color pixel respectively. A color difference generation process that is generated based on the second color component, and the first color component that the first color pixel that is the target for generating the second color difference has the second color difference. , One second color pixel of the second color pixel located above and below the first color pixel within the second range and the first color within the second range A color difference generation process that is generated based on the second color component of each of the second color pixels of the second color pixel located on the left and right of the pixel, and the second color difference The second color component of the second color pixel located above, below, left and right of the first color pixel that is the target for generating the second color difference, and the second color component. The other first color within the range and located on the opposite side of the second color pixel from the first color pixel A color difference generation process generated based on the first color component of the pixel, and performing at least two of the color difference generation processes, thereby at least two of the second color differences of the first color pixel. Is generated.
  An image processing apparatus according to a second aspect of the invention includes a first color pixel having a spectral sensitivity of a first color component and a second color component having a spectral sensitivity of a second color component different from the first color component. An image input unit that acquires image data output from an image sensor in which color pixels are arranged, and a first color difference of the first color pixel in a first range including the first color pixel A first color difference generation unit that generates by performing smoothing, and a second color difference that the first color pixel has is a second color difference that is smaller than the first range and includes the first color pixel. A second color difference generation unit configured to generate based on the second color component of the second color pixel located in the range, and the first color pixel based on the distribution range of the second color difference value. An output image based on the color difference having and the second color component of the second color pixel. And the second color difference generation unit includes the first color component included in the first color pixel that is a target for generating the second color difference. A color difference generation process that is generated based on the second color component of each of the second color pixels that are located in the second range and above and below the first color pixel, and The first color component of the first color pixel that is the target for generating the second color difference and the left and right of the first color pixel within the second range. The color difference generation process that is generated based on the second color component that each of the second color pixels that are positioned has, and the second color difference is the target for generating the second color difference. The first color component of the color pixel, and the upper and lower sides of the first color pixel within the second range. One second color pixel of the second color pixel located and one second color pixel of the second color pixel located on the left and right of the first color pixel within the second range The color difference generation process that is generated based on the second color component that each pixel has, and the second color difference is within the second range and is the target for generating the second color difference. The second color component of the second color pixel located on the top, bottom, left, and right of the first color pixel, and the second color component within the second range, the first color pixel with respect to the second color pixel By performing at least two color difference generation processes among the color difference generation processes generated based on the first color components of the other first color pixels located on the opposite side of the color pixels, At least two second color differences of the first color pixel are generated.
  An image processing program according to a third aspect of the invention includes a first color pixel having a spectral sensitivity of a first color component and a second color component having a spectral sensitivity of a second color component different from the first color component. Image input processing for acquiring image data output from an image sensor in which color pixels are arranged, and a first color difference of the first color pixel in a first range including the first color pixel A first color difference generation process generated by performing smoothing and a second color difference of the first color pixel, the second color difference including the first color pixel and smaller than the first range. A second color difference generation process that is generated based on the second color component of the second color pixel located in the range; and the first color difference value is in a distribution range of the second color difference value. If included, the first color difference is included, and the distribution range includes the first color difference value. If not, a color difference smoothing control process for generating a boundary value of the distribution range as a third color difference of the first color pixel, the second color component of the second color pixel, An image generation process for generating an output image based on the third color difference of the first color pixel; and the second color difference generation process includes: The first color component of the first color pixel, which is a target for generating a second color difference, and the second color located above and below the first color pixel in the second range. The color difference generation process generated based on the second color component each pixel has, and the first color pixel that is the target for generating the second color difference, the second color difference Color components and within the second range and to the left and right of the first color pixel The second color difference is generated on the basis of the second color component of each of the second color pixels to be placed, and the second color difference is the target for generating the second color difference. The first color component of the color pixel, the second color pixel of the second color pixel located above and below the first color pixel in the second range, and the second color pixel A color difference generation process that is generated on the basis of the second color component of each of the second color pixels of the second color pixel that is within the range and located on the left and right of the first color pixel; The second color difference is included in the second color pixel that is located in the second range and is located above, below, left, and right of the first color pixel that is the target for generating the second color difference. A second color component and the first color image within the second range with respect to the second color pixel. By performing at least two color difference generation processes among the color difference generation processes generated based on the first color components of the other first color pixels located on the opposite side of the element, At least two second color differences of the first color pixel are generated.
  An image processing program according to a fourth aspect of the invention includes a first color pixel having a spectral sensitivity of a first color component and a second color component having a spectral sensitivity of a second color component different from the first color component. Image input processing for acquiring image data output from an image sensor in which color pixels are arranged, and a first color difference of the first color pixel in a first range including the first color pixel A first color difference generation process generated by performing smoothing and a second color difference of the first color pixel, the second color difference including the first color pixel and smaller than the first range. A second color difference generation process generated based on the second color component of the second color pixel located in the range, and the first color pixel based on the distribution range of the second color difference value. Based on the color difference having and the second color component of the second color pixel. And an image generation process for generating an output image. The second color difference generation process is configured such that the second color difference is determined by the first color pixel that is a target for generating the second color difference. The first color component is generated based on the first color component, and the second color component included in each of the second color pixels located above and below the first color pixel in the second range. A color difference generation process; and the second color difference, the first color component of the first color pixel that is a target for generating the second color difference, and the second color difference within the second range and the first color component. A color difference generation process that is generated based on the second color components respectively included in the second color pixels located on the left and right of one color pixel, and the second color difference is generated as the second color difference. The first color component of the first color pixel that is the object; and the second color component. One second color pixel of the second color pixel located above and below the first color pixel and within the second range and located to the left and right of the first color pixel A color difference generation process that is generated based on the second color component of each of the second color pixels of the second color pixel, and the second color difference is within the second range. The second color component of the second color pixel located above, below, left and right of the first color pixel that is the target for generating the second color difference, and within the second range, A color difference generation process that is generated based on the first color component of the other first color pixel located on the opposite side of the first color pixel with respect to the second color pixel; By performing at least two color difference generation processes, the at least two previous color pixels have The second color difference is generated.

  ADVANTAGE OF THE INVENTION According to this invention, the fall of the saturation by color difference smoothing and a color blur can be suppressed, expanding the reference range of color difference smoothing.

It is a figure which shows the structure of the camera which outputs the raw image input into the image processing apparatus and image processing apparatus in one embodiment of this invention. It is a flowchart which shows the image processing in one embodiment of this invention. It is a figure which shows the pixel structural example of the RAW image used in order to demonstrate a brightness | luminance production | generation. It is a figure which shows the pixel structural example of the RAW image used in order to demonstrate color difference production | generation. It is a figure which shows an example of the smoothing filter used in order to demonstrate smoothing color difference production | generation. It is a figure which shows the wide pixel reference range referred at the time of smoothing color difference production | generation on a RAW image, and the local pixel reference range referred at the time of local color difference production | generation. FIG. 6 is a diagram for explaining a method of determining a color difference upper limit maxCr and a color difference lower limit minCr. It is a figure which shows the example of application of the image processing in one embodiment of this invention. It is a figure which shows the example of application of the image processing in one embodiment of this invention. It is a figure which shows the example of application of the image processing in one embodiment of this invention. It is a figure which shows the example of application of the image processing in one embodiment of this invention. FIG. 11 is a diagram illustrating a configuration of a camera incorporating an image processing apparatus as a modification of the present invention. It is a figure which shows the provision form of the computer program product for the image processing apparatus by this invention to perform an image process.

  FIG. 1 is a diagram illustrating a configuration of an image processing apparatus 100 and a camera 200 that outputs a RAW image input to the image processing apparatus 100 according to an embodiment of the present invention. The camera 200 includes a lens 210, an image sensor 220, and an original image reading unit 230. The lens 210 transmits the subject light 510 coming from the subject 500 and forms an optical image of the subject on the image sensor 220. In accordance with the imaging control by the original image reading unit 230, the image sensor 220 photoelectrically converts each color component of the G component, the R component, and the B component of the subject light 510 according to the spectral sensitivity of each color component. Each color component value is converted into a color component value and a B component value, and each color component value is output. In this way, the RAW image data of the RAW image (original image) is generated and output from the image sensor 220. For example, as shown in FIG. 3, the RAW image 300 includes a GR pixel column in which G pixels having a G component spectral sensitivity and R pixels having an R component spectral sensitivity are arranged in the horizontal direction (x direction). A BG pixel array in which B pixels having a B component spectral sensitivity and G pixels having a G component spectral sensitivity are arranged in the horizontal direction (x direction) and juxtaposed in the vertical direction (y direction) Direction), and GR pixel rows and BG pixel rows are alternately juxtaposed in the vertical direction (y direction).

  The image processing apparatus 100 is a computer that executes an image processing program, for example, a PC. The image processing apparatus 100 functionally includes an image input unit 1010, a smoothed color difference generation unit 1020, a local color difference generation unit 1030, a color difference smoothing control unit 1040, and an image generation unit 1050 by executing the image processing program. . The image input unit 1010, the smoothed color difference generation unit 1020, the local color difference generation unit 1030, the color difference smoothing control unit 1040, and the image generation unit 1050 perform each step of the image processing illustrated in FIG.

  FIG. 2 is a flowchart showing image processing performed in the image processing apparatus 100 according to the present embodiment. In step S <b> 200, the image input unit 1010 acquires RAW image data of the RAW image 300 captured by the image sensor 220 including, for example, a Bayer array color filter array and output by the original image reading unit 230. Hereinafter, description will be made assuming that the array of the RAW image 300 is the Bayer array.

In step S210, the image generation unit 1050 generates luminance by any known method. For example, the luminance Y may be generated for each pixel by the following method. That is, for each G pixel position (x, y) shown in FIG. 3, the luminance Y is equal to the spectral sensitivity of the G component of the G pixel, as shown in Equation (1). For each RB pixel position (x, y) shown in FIG. 4, | G (x−1, y) −G (x + 1, y) | + th <| G (x, y−1) −G ( In the case of x, y + 1) |, the luminance Y is generated by equation (2), and | G (x-1, y) -G (x + 1, y) |> | G (x, y-1) In the case of -G (x, y + 1) | + th, the luminance Y is generated by the equation (3), and in any other case, the luminance Y is generated by the equation (4). However, the value of the parameter th is about three times the standard deviation of the noise included in the G pixel value.
Y (x, y) = G (x, y) (1)
Y (x, y) = (G (x-1, y) + G (x + 1, y)) / 2 (2)
Y (x, y) = (G (x, y-1) + G (x, y + 1)) / 2 (3)
Y (x, y) = (G (x-1, y) + G (x + 1, y) + G (x, y-1) + G (x, y + 1)) / 4 (4)

  In step S220, the smoothed color difference generation unit 1020 generates a color difference intermediate value by a known method at each RB pixel position (x, y). In this processing, it is determined whether the image structure is vertical or horizontal by a known method, and an intermediate value of the color difference is generated by referring to the pixel in the direction corresponding to the result.

  First, the direction may be determined as follows, for example. That is, if | G (x-1, y) -G (x + 1, y) | + th <| G (x, y-1) -G (x, y + 1) | | G (x-1, y) -G (x + 1, y) |> | G (x, y-1) -G (x, y + 1) | + th In this case, it is determined that the image structure is “vertical direction”, and when it is neither of them, it is determined that the image structure is “neither”.

When the pixel arranged at the pixel position (x, y) is an R pixel, the smoothing color difference generation unit 1020 uses the following expressions (5) to (7) according to the image structure to calculate the pixel position (x, An intermediate value Cr1 of the color difference in y) is generated. When it is determined that the image structure is “lateral direction”, the intermediate value Cr1 of the color difference is generated by the equation (5). When it is determined that the image structure is “vertical direction”, the intermediate value Cr1 of the color difference is generated by Expression (6). When it is determined that the image structure is “neither”, the color difference intermediate value Cr1 is generated by Expression (7).
Cr1 = R− (G (x-1, y) + G (x + 1, y)) / 2 (5)
Cr1 = R− (G (x, y-1) + G (x, y + 1)) / 2 (6)
Cr1 = R- (G (x-1, y) + G (x + 1, y) + G (x, y-1) + G (x, y + 1)) / 4 (7)

  When the pixel arranged at the pixel position (x, y) is a B pixel, the processing may be performed by replacing R with B and Cr with Cb. Thus, the smoothed color difference generation unit 1020 generates the color difference intermediate value Cb1 at the pixel position (x, y) according to the image structure.

Next, the smoothed color difference generation unit 1020 smoothes the color difference intermediate values Cr1 and Cb1 by a known method at each RB pixel position (x, y) to generate the smoothed color differences CrS and CbS. This processing may be performed as follows, for example. When the pixel arranged at the pixel position (x, y) shown in FIG. 4A is an R pixel, the weighted smoothing filter shown in FIG. The smoothed color difference CrS of y) is generated. Pixel position (x-4, y), (x-2) where R pixel at smoothing color difference generation target pixel position (x, y) and two R pixels on both sides in the horizontal direction (on the same line) are arranged , y), (x + 2, y), (x + 4, y), a weighted smoothing filter F3 = (1, 2, 2, 2, as shown in FIG. By applying 1), an intermediate value tmp3 corresponding to the y-th row is obtained. This is expressed by equation (10). Similarly, two R pixels at the pixel position (x, y-4) on the four rows of the R pixel at the smoothed color difference generation target pixel position (x, y) and two sides in the horizontal direction (on the same line). Color difference at pixel positions (x-4, y-4), (x-2, y-4), (x + 2, y-4), (x + 4, y-4) where the R pixels are arranged By applying a weighted smoothing filter F1 = (1, 2, 2, 2, 1) as shown in FIG. 5 to the intermediate value Cr1, the R of the pixel position (x, y) to be smoothed color difference generation An intermediate value tmp1 corresponding to the (y-4) th row above the fourth row of pixels is obtained. This is expressed by equation (8). The R pixel at the pixel position (x, y-2) on the two rows of the R pixel at the smoothed color difference generation target pixel position (x, y) and two R pixels on both sides in the horizontal direction (on the same line) Intermediate value Cr1 of the color difference at the arranged pixel position (x-4, y-2), (x-2, y-2), (x + 2, y-2), (x + 4, y-2) By applying a weighting filter F2 = (1, 2, 2, 2, 1) as shown in FIG. 5, two rows above the R pixel at the smoothed color difference generation target pixel position (x, y). (Y-2) An intermediate value tmp2 corresponding to the line is obtained. This is expressed by equation (9). The R pixel at the pixel position (x, y + 2) two rows below the R pixel at the smoothed color difference generation target pixel position (x, y) and two R pixels on both sides in the horizontal direction (on the same line) Intermediate value Cr1 of color difference at the arranged pixel position (x-4, y + 2), (x-2, y + 2), (x + 2, y + 2), (x + 4, y + 2) By applying a weighting filter F4 = (1, 2, 2, 2, 1) as shown in FIG. 5, two rows below the R pixel at the smoothed color difference generation target pixel position (x, y) An intermediate value tmp4 corresponding to the (y + 2) line is obtained. This is expressed by equation (11). The R pixel at the pixel position (x, y + 4) four rows below the R pixel at the smoothed color difference generation target pixel position (x, y) and two R pixels on both sides in the horizontal direction (on the same line) Intermediate value Cr1 of the color difference at the arranged pixel position (x-4, y + 4), (x-2, y + 4), (x + 2, y + 4), (x + 4, y + 4) By applying a weighted smoothing filter F5 = (1, 2, 2, 2, 1) as shown in FIG. 5, four rows of R pixels at the smoothed color difference generation target pixel position (x, y) An intermediate value tmp5 corresponding to the lower (y + 4) line is obtained. This is expressed by equation (12).
tmp1 = Cr1 (x-4, y-4) + 2 * Cr1 (x-2, y-4) + 2 * Cr1 (x, y-4) + 2 * Cr1 (x + 2, y-4) + Cr1 (x + 4, y-4)
(8)
tmp2 = Cr1 (x-4, y-2) + 2 * Cr1 (x-2, y-2) + 2 * Cr1 (x, y-2) + 2 * Cr1 (x + 2, y-2) + Cr1 (x + 4, y-2)
(9)
tmp3 = Cr1 (x-4, y) + 2 * Cr1 (x-2, y) + 2 * Cr1 (x, y) + 2 * Cr1 (x + 2, y) + Cr1 (x + 4, y)
(10)
tmp4 = Cr1 (x-4, y + 2) + 2 * Cr1 (x-2, y + 2) + 2 * Cr1 (x, y + 2) + 2 * Cr1 (x + 2, y + 2) + Cr1 (x + 4, y + 2)
(11)
tmp5 = Cr1 (x-4, y + 4) + 2 * Cr1 (x-2, y + 4) + 2 * Cr1 (x, y + 4) + 2 * Cr1 (x + 2, y + 4) + Cr1 (x + 4, y + 4)
(12)

The intermediate values tmp1 to tmp5 thus determined are multiplied by weights (W1, W2, W3, W4, W5) = (1, 2, 2, 2, 1) shown in FIG. The smoothed color difference CrS is obtained by dividing the added total value by the weighted total value 64 of the weighted smoothing filters F1 to F5 and the weights W1 to W5 as a whole, that is, by Expression (13). Thus, the smoothed color difference generation unit 1020 calculates the smoothed color difference CrS at the pixel position (x, y) when the pixel arranged at the pixel position (x, y) shown in FIG. It is generated by performing smoothing in a wide pixel reference range SF of 9 pixels × 9 pixels including R pixels arranged at the pixel position (x, y) as shown in FIG. The wide pixel reference range SF of 9 pixels × 9 pixels has 81 pixel positions (x-4, y-4), (x-3, y-4), (x-2) as shown in FIG. , y-4), (x-1, y-4), (x, y-4), (x + 1, y-4), (x + 2, y-4), (x + 3, y -4), (x + 4, y-4), (x-4, y-3), (x-3, y-3), (x-2, y-3), (x-1, y -3), (x, y-3), (x + 1, y-3), (x + 2, y-3), (x + 3, y-3), (x + 4, y-3 ), (X-4, y-2), (x-3, y-2), (x-2, y-2), (x-1, y-2), (x, y-2), (x + 1, y-2), (x + 2, y-2), (x + 3, y-2), (x + 4, y-2), (x-4, y-1), (x-3, y-1), (x-2, y-1), (x-1, y-1), (x, y-1), (x + 1, y-1), (x + 2, y-1), (x + 3, y-1), (x + 4, y-1), (x-4, y), (x-3, y), (x-2, y ), (X-1, y), (x, y), (x + 1, y), (x + 2, y), (x + 3, y), (x + 4, y), (x -4, y + 1), (x-3, y + 1), (x-2, y + 1), (x-1, y + 1), (x, y + 1), (x + 1 , y + 1), (x + 2, y + 1), (x + 3, y + 1), (x + 4, y + 1), (x-4, y + 2), (x-3 , y + 2), (x-2, y + 2), (x-1, y + 2), (x, y + 2), (x + 1, y + 2), (x + 2, y +2), (x + 3, y + 2), (x + 4, y + 2), (x-4, y + 3), (x-3, y + 3), (x-2, y +3), (x-1, y + 3), (x, y + 3), (x + 1, y + 3), (x + 2, y + 3), (x + 3, y + 3 ), (X + 4, y + 3), (x-4, y + 4), (x-3, y + 4), (x-2, y + 4), (x-1, y + 4) ), (X, y + 4) (X + 1, y + 4), it contains (x + 2, y + 4), (x + 3, y + 4), and (x + 4, y + 4).
Smoothing color difference CrS (x, y) = (tmp1 + 2 * tmp2 + 2 * tmp3 + 2 * tmp4 + tmp5) / 64 (13)

  When the pixel arranged at the pixel position (x, y) is a B pixel, the above Cr may be replaced with Cb. Thus, the smoothed color difference generation unit 1020 calculates the smoothed color difference CbS at the pixel position (x, y) when the pixel arranged at the pixel position (x, y) shown in FIG. It is generated by performing smoothing in a wide pixel reference range of 9 pixels × 9 pixels including the B pixel arranged at the pixel position (x, y).

  In step S230, the local color difference generation unit 1030 generates local color differences CrA to CrF and CbA to CbF at each RB pixel position (x, y). The local color difference CrA of the R pixel at the pixel position (x, y) is generated based on the R component of the R pixel and the G component of the G pixel located in the local pixel reference range LF in the vicinity of the R pixel. Is done. The local pixel reference range LF includes pixel positions (x, y), (x-1, y), (x + 1, y), (x, y-1), and (x, y + 1). It is. The local pixel reference range LF used for generating the local color differences CrA to CrF is smaller than the wide-area pixel reference range SF used for generating the smoothed color difference CrS. Similarly, the local color difference CbA of the B pixel at the pixel position (x, y) is based on the B component of the B pixel and the G component of the G pixel located in the local pixel reference range near the B pixel. Generated.

When the pixel arranged at the pixel position (x, y) is an R pixel, the local color difference generation unit 1030, for example, as represented by the following equations (14) to (19), the pixel position (x, 6 patterns of local color differences CrA to CrF possessed by the R pixel arranged in y) are adjacent to the R pixel within the R component of the R pixel and the local pixel reference range LF in the vicinity of the R pixel. It is generated by performing color difference generation processes A to F based on the G component each of the two G pixels has. In the color difference generation process A shown in Expression (14), the local color difference CrA is calculated by using the R component of the R pixel arranged at the pixel position (x, y) and the right and left of the R pixel within the local pixel reference range LF. Is generated based on the G component of each of the G pixels at pixel positions (x−1, y) and (x + 1, y) adjacent to. In the color difference generation process B shown in Expression (15), the local color difference CrB is calculated by using the R component of the R pixel arranged at the pixel position (x, y) and the upper and lower of the R pixel within the local pixel reference range LF. Are generated based on the G components of the G pixels at the pixel positions (x, y-1) and (x, y + 1) adjacent to. In the color difference generation process C shown in Expression (16), the local color difference CrC is calculated by using the R component of the R pixel arranged at the pixel position (x, y) and the local pixel reference range LF above the R pixel. And the G components of the G pixels at the pixel positions (x, y-1) and (x-1, y) adjacent to the left are generated. In the color difference generation processing D shown in Expression (17), the local color difference CrD is calculated by using the R component of the R pixel arranged at the pixel position (x, y) and the local pixel reference range LF above the R pixel. And the G components of the G pixels at the pixel positions (x, y−1) and (x + 1, y) adjacent to the right are generated. In the color difference generation process E shown in the equation (18), the local color difference CrE is calculated based on the R component of the R pixel arranged at the pixel position (x, y) and the local pixel reference range LF below the R pixel. And the G components of the G pixels at the pixel positions (x, y + 1) and (x-1, y) adjacent to the left are generated. In the color difference generation process F shown in the equation (19), the local color difference CrF is calculated based on the R component of the R pixel arranged at the pixel position (x, y) and the local pixel reference range LF below the R pixel. And the G components of the G pixels at the pixel positions (x, y + 1) and (x + 1, y) adjacent to the right, respectively.
Color difference generation processing A: CrA (x, y) = R (x, y) − (G (x−1, y) + G (x + 1, y)) / 2 (14)
Color difference generation processing B: CrB (x, y) = R (x, y) − (G (x, y−1) + G (x, y + 1)) / 2 (15)
Color difference generation processing C: CrC (x, y) = R (x, y) − (G (x, y−1) + G (x−1, y)) / 2 (16)
Color difference generation processing D: CrD (x, y) = R (x, y) − (G (x, y−1) + G (x + 1, y)) / 2 (17)
Color difference generation processing E: CrE (x, y) = R (x, y) − (G (x, y + 1) + G (x−1, y)) / 2 (18)
Color difference generation processing F: CrF (x, y) = R (x, y) − (G (x, y + 1) + G (x + 1, y)) / 2 (19)

  When the pixel arranged at the pixel position (x, y) is a B pixel, the above Cr may be replaced with Cb. In this way, the local color difference generation unit 1030 includes the six local color differences CbA to CbF of the B pixel arranged at the pixel position (x, y), the B component of the B pixel, and the local pixels in the vicinity of the B pixel. It is generated by performing six patterns of color difference generation processing based on the G component that each of the two G pixels adjacent to the B pixel within the reference range has.

In step S235, the color difference smoothing control unit 1040 performs the following processing, for example, at each RB pixel position (x, y) to generate image generation color differences Cr3 and Cb3. That is, when the R pixel is arranged at the pixel position (x, y), if the smoothed color difference CrS (x, y) <the color difference lower limit minCr (x, y), the image generation is performed according to the equation (20). Color difference Cr3 (x, y) is generated. That is, the color difference lower limit minCr (x, y) is generated as the image generation color difference Cr3 (x, y). If smooth color difference CrS (x, y)> color difference upper limit maxCr (x, y), the color difference Cr3 (x, y) for image generation is generated by Expression (21). That is, the color difference upper limit maxCr (x, y) is generated as the image generation color difference Cr3 (x, y). Otherwise, the image generation color difference Cr3 (x, y) is generated according to the equation (22). That is, the smoothed color difference CrS (x, y) is generated as the image generation color difference Cr3 (x, y). Note that the color difference lower limit minCr (x, y) and the color difference upper limit maxCr (x, y) are respectively generated by the local color difference generation unit 1030 in step S230 and the local color difference CrA (at each R pixel position (x, y). x, y) to the boundary value of the distribution range of values of CrF (x, y). When the R pixel is arranged at the pixel position (x, y), the R is in the distribution range of values of the local color difference CrA (x, y) to CrF (x, y) at the R pixel position (x, y). When the pixel smoothing color difference CrS (x, y) is included, the color difference smoothing control unit 1040 generates the smoothing color difference CrS (x, y) as the image generation color difference Cr3 (x, y). When the local color difference CrA (x, y) to CrF (x, y) distribution range of the R pixel position (x, y) does not include the smoothed color difference CrS (x, y) of the R pixel, The color difference smoothing control unit 1040 generates the boundary value of the distribution range of the values of the local color differences CrA (x, y) to CrF (x, y) as the image generation color difference Cr3 (x, y). One of the boundary values, the color difference lower limit minCr (x, y), is the minimum value of the distribution range, and the other boundary value, the color difference upper limit maxCr (x, y), is the maximum value of the distribution range. For example, in the example shown in FIG. 7, the minimum value and the maximum value of CrA to CrF by the six patterns of color difference generation processing are the local color difference CrB value and the local color difference CrC value, respectively.
Cr3 (x, y) = minCr (x, y) (20)
Cr3 (x, y) = maxCr (x, y) (21)
Cr3 (x, y) = CrS (x, y) (22)

  When the pixel arranged at the pixel position (x, y) is a B pixel, the above Cr may be replaced with Cb. If the distribution range of the local color difference CbA (x, y) to CbF (x, y) at the B pixel position (x, y) includes the smoothed color difference CbS (x, y) of that B pixel, the color difference The smoothing control unit 1040 generates the smoothed color difference CbS (x, y) as the image generation color difference Cb3 (x, y). When the smooth color difference CbS (x, y) of the B pixel is not included in the distribution range of values of the local color difference CbA (x, y) to CbF (x, y) at the B pixel position (x, y), The color difference smoothing control unit 1040 generates the boundary value of the distribution range of the values of the local color differences CbA (x, y) to CbF (x, y) as the image generation color difference Cb3 (x, y). One of the boundary values, the color difference lower limit minCb (x, y) is the minimum value of the distribution range, and the other boundary value, the color difference upper limit maxCb (x, y), is the maximum value of the distribution range.

  In step S250, the image generation unit 1050 generates luminance at all pixel positions generated based on the G component of the G pixel in step S210. In step S235, the image generation color difference Cr3 (x, y) of the R pixel. ) To generate an image generation color difference Cr3 at all pixel positions, and generate an image generation color difference Cb3 at all pixel positions based on the image generation color difference Cb3 (x, y) of the B pixel. That is, the image generation unit 1050 interpolates and generates image generation color differences Cr3 and Cb3 for all pixels, uses the luminance of all pixels generated in step S210, and uses G = Y, R = Y + Cr, B for all pixels. RGB components can be generated by setting = Y + Cb. The image generation unit 1050 outputs the generated RGB output image after further performing known image processing such as saturation enhancement and contour enhancement.

  Based on a specific example, it will be described that the image processing apparatus 100 according to the present embodiment can suppress adverse effects caused by color difference smoothing. FIG. 8 shows a RAW image 300 having a red region 80 obtained by imaging a red subject on the right side of the R pixel position (x, y) to be subjected to color difference smoothing surrounded by a circular frame. In FIG. 8, the red region 80 is illustrated by hatching. The red area 80 is a red flat area, and the other areas are white flat areas. The wide pixel reference range SF at the time of smoothing by the smoothing color difference generation unit 1020 is a range of 9 pixels × 9 pixels centering on the R pixel position (x, y) to be subjected to color difference smoothing. It is wider than 5 pixels × 5 pixels that are normally used in the known technology as disclosed in Japanese Patent No. The original color difference at the pixel position (x, y) belonging to the white flat area is 0, but the wide area pixel reference range SF at the time of smoothing by the smoothing color difference generation unit 1020 includes many pixel positions belonging to the red area 80. Therefore, the smoothed color difference CrS (x, y) becomes a large value. On the other hand, since the local pixel reference range LF at the time of local color difference generation by the local color difference generation unit 1030 includes only a white flat region, the local color differences CrA (x, y) to CrF (x, y) are almost zero. . Accordingly, the color difference lower limit minCr (x, y) and the color difference upper limit maxCr (x, y) are also substantially zero. Then, the color difference smoothing control unit 1040 limits the smoothed color difference CrS (x, y) to a range between the color difference lower limit minCr (x, y) and the color difference upper limit maxCr (x, y), thereby generating an image. The color difference of the color difference Cr3 (x, y) is also almost zero. In this way, even when the wide pixel reference range SF at the time of smoothing by the smoothing color difference generation unit 1020 is made wider than before, a color difference with suppressed color bleeding can be generated.

  Next, it will be described that the false color can be appropriately removed according to the image processing apparatus 100 in the present embodiment. First, the false color removal of a subject having a black and white vertical stripe structure slightly inclined as shown in FIG. 9 will be described. As shown in FIG. 9, the vertical and horizontal directions are determined based on the Bayer array RAW image 300 in the case of a two-pixel periodic structure stripe pattern in which vertical black stripes 90 appear slightly every other pixel in the horizontal direction on a white background. It is difficult. Therefore, when the smoothed color difference generation unit 1020 generates the color difference intermediate value Cr1 (x, y), it is determined that the image structure is “neither”, and Expression (7) is used. According to Equation (7), for example, at a color difference generation location represented by a pixel position (x, y) surrounded by a circular frame, a pixel including a part of the vertical stripe 90 (a pixel in the local pixel reference range LF and Is equal), the term of G becomes dark and false color occurs. On the other hand, in the pixel positions (x, y-4) and (x, y + 4) that are 4 pixels away from the pixel position (x, y) surrounded by the circular frame, A false color of the code occurs. That is, the false colors cancel each other out within the wide pixel reference range SF at the time of smoothing by the smoothing color difference generation unit 1020, and the smoothing color difference CrS (x, y) becomes a small value.

  When the local color difference CrB (x, y) is generated by the equation (14) among the plurality of color differences generated by the local color difference generation unit 1030, the pixel positions (x, y), (x, y-1) are generated. ) And (x, y + 1) are referred to as a substantially flat range LFB, so that the local color difference CrB (x, y) is almost zero. When generating local color differences CrA (x, y), CrC (x, y), CrD (x, y), CrE (x, y) and CrF (x, y) other than local color difference CrB (x, y) Since they refer to dark G, their local color differences are all large. Accordingly, the local color difference CrB (x, y) is selected as the color difference lower limit minCr (x, y). Since the smoothed color difference CrS (x, y) is larger than the color difference lower limit minCr (x, y), that is, the local color difference CrB (x, y) that becomes almost zero, it is smoothed as the color difference Cr3 (x, y) for image generation. Color difference CrS (x, y) is adopted, and false colors are removed.

  Next, false color removal when there is a dark region 91 having an oblique outline as shown in FIG. 10 will be described. In this case, when the color difference intermediate value Cr1 (x, y) is generated by the smoothing color difference generation unit 1020, it is determined that the image structure is “neither”, and Expression (7) is used. According to Expression (7), for example, at a place represented by a pixel position (x, y) surrounded by a circular frame, a pixel including a part of the dark region 91 (equal to a pixel in the local pixel reference range LF) ), The term of G becomes dark and false color occurs. On the other hand, since most of the color differences in the wide pixel reference range SF at the time of smoothing by the smoothing color difference generation unit 1020 are 0, the smoothing color difference CrS (x, y) is a small value.

  Of the plurality of color differences generated by the local color difference generation unit 1030, when the local color difference CrE (x, y) is generated by the equation (18), the pixel position (x, y), (x-1, y) is generated. ) And (x, y + 1) are referred to, and the local color difference CrE (x, y) is almost zero. When generating local color differences CrA (x, y), CrB (x, y), CrC (x, y), CrD (x, y) and CrF (x, y) other than local color difference CrE (x, y) Since they refer to dark G, their local color differences are large. Therefore, the local color difference CrE (x, y) is selected as the color difference lower limit minCr (x, y). Since the smoothed color difference CrS (x, y) is larger than the color difference lower limit minCr (x, y), that is, the local color difference CrE (x, y) that becomes almost zero, the smoothed color difference as the color difference Cr3 (x, y) for image generation CrS (x, y) is adopted to remove false colors.

  The image processing apparatus 100 according to the present embodiment includes an image input unit 1010, a smoothed color difference generation unit 1020, a local color difference generation unit 1030, a color difference smoothing control unit 1040, and an image generation unit 1050. The image input unit 1010 includes a RAW image 300 in which a pixel row in which R pixels having an R component spectral sensitivity or B pixels having a B component spectral sensitivity and G pixels having a G component spectral sensitivity are arranged side by side is arranged. , RAW image data is acquired. The smoothing color difference generation unit 1020 uses the smoothing color difference CrS (x, y) of the R pixel or the smoothing color difference CbS (x, y) of the B pixel in the wide pixel reference range including the R pixel or B pixel. Generated by performing smoothing. The local color difference generation unit 1030 converts the local color differences CrA (x, y) to CrF (x, y) included in the R pixel or the local color differences CbA (x, y) to CbF (x, y) included in the B pixel into the R pixel. Or it produces | generates based on R component or B component which R pixel or B pixel located in a local pixel reference range rather than a wide pixel reference range of the vicinity of B pixel has. The color difference smoothing control unit 1040, when the value of the smoothed color difference CrS (x, y) or CbS (x, y) is included in the distribution range of the local color difference value, the smoothed color difference CrS (x, y) or CbS. (x, y), if the distribution range of the local color difference value does not include the smoothing color difference CrS (x, y) or CbS (x, y), the boundary value of the distribution range is set to R pixels or It is generated as a color difference Cr3 (x, y) or Cb3 (x, y) for image generation of the B pixel. The image generation unit 1050 generates an output image based on the G component of the G pixel and the image generation color difference Cr3 (x, y) or Cb3 (x, y) of the R pixel or B pixel. Accordingly, it is possible to generate an output image in which color bleeding is suppressed while removing false colors.

The image processing apparatus 100 according to the embodiment described above may be modified as follows.
(1) The local color difference generation unit 1030 uses the following equation (23) in addition to the local color differences CrA (x, y) to CrF (x, y) corresponding to the R pixel at the pixel position (x, y). A local color difference CrG (x, y) may be further generated by performing the color difference generation process G.
CrG (x, y) = (R (x-2, y) + R (x, y-2) + R (x, y + 2) + R (x + 2, y)) / 4
− (G (x−1, y) + G (x, y−1) + G (x, y + 1) + G (x + 1, y)) / 4 (23)

  In this case, the color difference smoothing control unit 1040 sets the maximum value of the local color differences CrA (x, y) to CrG (x, y) as the color difference upper limit maxCr (x, y) at each R pixel position (x, y). The minimum value of local color differences CrA (x, y) to CrG (x, y) is defined as a color difference lower limit minCr (x, y). The same applies to the local color difference Cb at the B pixel position. An example of the application will be described with reference to FIG.

  FIG. 11 is an example of a RAW image 300 having a dark thin diagonal line 92 at a pixel position in one diagonal RB pixel row. In the case of the color difference intermediate value Cr1 by the smoothing color difference generation unit 1020, it is determined that the image structure is “neither”, and Expression (7) is used. According to Expression (7), for example, the intermediate value Cr1 of the color difference at the pixel position (x, y) surrounded by the circular frame in FIG. 11 is generated by referring to the same pixel as the pixel in the local pixel reference range LF. The Since the R at the pixel position (x, y) included in the dark thin diagonal line 92 is small, a negative false color is generated. On the other hand, since most of the color differences in the wide pixel reference range SF at the time of smoothing by the smoothing color difference generation unit 1020 are 0, the smoothing color difference CrS (x, y) is a small value.

  When the local color difference CrG (x, y) is generated by the equation (23) among the plurality of color differences generated by the local color difference generation unit 1030, a flat reference range LFga obtained by extending the local pixel reference range LF, Since LFgb, LFgc, and LFgd are referenced, the local color difference CrG (x, y) is almost zero. Local color difference other than local color difference CrG (x, y) CrA (x, y), CrB (x, y), CrC (x, y), CrD (x, y), CrE (x, y) and CrF (x , y) is generated, since dark R is referred to, their local color difference becomes a negative value. Therefore, the color difference CrG (x, y) is selected as the color difference upper limit maxCr (x, y). Since the smoothed color difference CrS (x, y) is larger than the color difference upper limit maxCr (x, y), that is, the local color difference CrG (x, y) that becomes almost zero, the smoothed color difference as the image generation color difference Cr3 (x, y) CrS (x, y) is adopted to remove false colors. The reference range LFga includes pixel positions (x−2, y) and (x−1, y). The reference range LFgb includes pixel positions (x, y-2) and (x, y-1). The reference range LFgc includes pixel positions (x, y + 2) and (x, y + 1). The reference range LFgc includes pixel positions (x + 2, y) and (x + 1, y). The same applies to the local color difference corresponding to the B pixel at the pixel position (x, y).

(2) The local color difference generation unit 1030 calculates the local color differences CrA and CrB, and the color difference smoothing control unit 1040 calculates the maximum value of the local color differences CrA and CrB at each R pixel position (x, y) as the color difference upper limit maxCr ( x, y), and the minimum value of the local color differences CrA, CrB may be used as the color difference lower limit minCr (x, y). The same applies to the local color difference Cb. By reducing the number of color differences calculated in this way, the distribution range of local color difference values is the same as or narrower than that of the above-described embodiment, so that the adverse effects of color difference smoothing are more strongly suppressed. . On the other hand, in the structure in which false color is particularly strong as shown in FIG. 9, there is a false color suppression effect similar to that of the above-described embodiment.

(3) The smoothing color difference generation unit 1020 may perform smoothing with reference to the color difference intermediate value Cr1 and the image generation color difference Cr3. For example, in the case where the processing of the present invention is performed in order from the pixel on the upper side of the RAW image 300 (the one with the smaller vertical coordinate y), when the pixel at the pixel position (x, y) is processed, The image generation color difference Cr3 is already generated at the position. Therefore, for example, when an R pixel is arranged at the pixel position (x, y), the color difference is smoothed by using equations (24) to (29) instead of equations (8) to (13). The color difference CrS can be calculated. If a B pixel is arranged at the pixel position (x, y), the local color difference Cr may be replaced with the local color difference Cb.
tmp1´ = Cr3 (x-4, y-4) + 2 * Cr3 (x-2, y-4) + 2 * Cr3 (x, y-4) + 2 * Cr3 (x + 2, y-4) + Cr3 (x + 4, y-4)
(24)
tmp2´ = Cr3 (x-4, y-2) + 2 * Cr3 (x-2, y-2) + 2 * Cr3 (x, y-2) + 2 * Cr3 (x + 2, y-2) + Cr3 (x + 4, y-2)
(25)
tmp3 = Cr (x-4, y) + 2 * Cr (x-2, y) + 2 * Cr (x, y) + 2 * Cr (x + 2, y) + Cr (x + 4, y)
(26)
tmp4 = Cr (x-4, y + 2) + 2 * Cr (x-2, y + 2) + 2 * Cr (x, y + 2) + 2 * Cr (x + 2, y + 2) + Cr (x + 4, y + 2)
(27)
tmp5 = Cr (x-4, y + 4) + 2 * Cr (x-2, y + 4) + 2 * Cr (x, y + 4) + 2 * Cr (x + 2, y + 4) + Cr (x + 4, y + 4)
(28)
CrS (x, y) = (tmp1´ + 2 * tmp2´ + 2 * tmp3 + 2 * tmp4 + tmp5) / 64
(29)

  That is, when the R pixel is arranged at the pixel position (x, y), the smoothing color difference generation unit 1020 has a wide area when the other R pixel located in the wide area pixel reference range SF has the image generation color difference Cr3. Smoothing color difference CrS is generated by performing smoothing using the image generation color difference Cr3 possessed by other R pixels located in the pixel reference range SF. When the B pixel is arranged at the pixel position (x, y), when another B pixel located in the wide pixel reference range SF has the image generation color difference Cb3, another B pixel located in the wide pixel reference range SF The smoothing color difference CbS is generated by performing smoothing using the image generation color difference Cb3 of the pixel.

  In this way, since smoothing is further performed with reference to the smoothed color difference Cr3 or Cb3 for image generation, this corresponds to a smoothing stronger than the above-described one embodiment, and false colors are It can be removed strongly. In addition, since the image generation color difference Cr3 or Cb3, in which the color bleeding control is suppressed by the color difference smoothing control unit 1040, is referred to, it is difficult for the color to ooze instead of performing strong color difference smoothing.

Further, the smoothed color difference CrS (x, y) may be calculated using equations (30) to (32) instead of equations (24) to (29).
tmp2´ = Cr3 (x-4, y-2) + 2 * Cr3 (x-2, y-2) + 2 * Cr3 (x, y-2) + 2 * Cr3 (x + 2, y-2) + Cr3 (x + 4, y-2)
(30)
tmp3 = Cr (x-4, y) + 2 * Cr (x-2, y) + 2 * Cr (x, y) + 2 * Cr (x + 2, y) + Cr (x + 4, y)
(31)
CrS (x, y) = (3 * tmp2´ + tmp3) / 32
(32)

  According to this calculation formula, the calculation cost can be reduced because the wide-area pixel reference range is narrower than in the above-described embodiment. Further, since the smoothing is performed with strong reference to the image generation color difference Cr3 from which the false color is removed, there is a strong false color removal effect. The color difference Cb may be the same as described above.

(4) Although the G component value of each pixel is used as the luminance Y in step S210, the weighted average value of the G component, R component, and B component of each pixel may be used as the luminance Y. In that case, the false color removal effect by the smoothing is also obtained for the G component.

(5) In step S235, the color difference smoothing control unit 1040 uses the boundary value (minimum value and maximum value) of the distribution range of the local color difference CrA (x, y) to CrF (x, y) as the color difference lower limit minCr (x , y) and the color difference upper limit maxCr (x, y). However, the color difference lower limit minCr (x, y) and the color difference upper limit maxCr (x, y) are determined based on the boundary value of the distribution range of values of the local color difference CrA (x, y) to CrF (x, y). For example, other values may be used. For example, values obtained by increasing or decreasing the boundary value by a predetermined value may be used as the color difference lower limit minCr (x, y) and the color difference upper limit maxCr (x, y). Alternatively, the next smallest value after the minimum value may be the color difference lower limit minCr (x, y), and the next largest value after the maximum value may be the color difference upper limit maxCr (x, y).

(6) The image processing apparatus 100 according to the embodiment described above is not built in the camera 200. Therefore, the RAW image captured by the image sensor 220 inside the camera 200 is acquired by the image input unit 1010 in the image processing apparatus 100 via the original image reading unit 230. However, the present invention can be applied even if the image processing apparatus 100 is built in the camera 200. FIG. 12 is a diagram showing a configuration of a camera 200 incorporating the image processing apparatus 100 as a modification of the present invention. The camera 200 is the same as that described in the above-described embodiment except that the camera 200 includes the control device 250 including the original image reading unit 230 and the image processing device 100. The image processing by the image processing apparatus 100 described with reference to the flowchart shown in FIG. 2 is managed by the control apparatus 250 as part of the control of the entire camera 200 by the control apparatus 250.

(7) As shown in FIG. 13, a computer program product for the image processing apparatus 100 to execute image processing in each of the above-described embodiments and modifications is a program storage medium 2000 (for example, a CD-ROM) or communication. It is provided to the image processing apparatus 100 through a data signal transmitted through a network 4000 (for example, the Internet).

  The image processing apparatus 100 is provided with an image processing program via the program storage medium 2000. Alternatively, the image processing program may be provided in the following supply form. The image processing apparatus 100 has a connection function with the communication network 4000. The server 3000 is a computer that provides an image processing program, and stores the image processing program in a storage medium such as a hard disk. The server 3000 places the image processing program read from the hard disk on a carrier wave as a data signal and transfers the image processing program to the image processing apparatus 100 via the communication network 4000. As described above, the image processing program is preferably supplied by various forms of computer program products including provision as data signals via the program storage medium 2000 and the communication network 4000.

80 red areas, 90 vertical stripes, 91 dark areas, 92 diagonal lines,
100 image processing device, 200 camera,
210 lens, 220 image sensor, 230 original image reading unit,
250 control device, 300 RAW image,
500 subjects, 510 subject light,
1010 Image input unit, 1020 smoothing color difference generation unit, 1030 local color difference generation unit,
1040 color difference smoothing control unit, 1050 image generation unit,
2000 program storage medium, 3000 server, 4000 communication network

Claims (8)

The first color pixel having the spectral sensitivity of the first color component and the second color pixel having the spectral sensitivity of the second color component different from the first color component are output from the imaging device arranged. An image input unit for acquiring the obtained image data;
A first color difference generation unit that generates the first color difference of the first color pixel by performing smoothing in a first range including the first color pixel;
The second color difference of the first color pixel is included in the second color pixel of the second color pixel that is located in a second range that is smaller than the first range and that includes the first color pixel. A second color difference generation unit that generates based on color components;
The first color difference value is included when the second color difference value distribution range includes the first color difference value, and the distribution range when the first color difference value is not included in the distribution range. A color difference smoothing control unit that generates a boundary value of the third color difference of the first color pixel;
An image generation unit configured to generate an output image based on the second color component of the second color pixel and the third color difference of the first color pixel ;
The second color difference generation unit includes:
The first color component of the first color pixel that is the target for generating the second color difference, the second color difference, and the first color pixel within the second range. A color difference generation process that is generated based on the second color components of the second color pixels positioned above and below,
The first color component of the first color pixel that is the target for generating the second color difference, the second color difference, and the first color pixel within the second range. A color difference generation process that is generated based on the second color components of the second color pixels located on the left and right, respectively,
The first color component of the first color pixel that is the target for generating the second color difference, the second color difference, and the first color pixel within the second range. One second color pixel of the second color pixel located above and below and the second one of the second color pixels located on the left and right of the first color pixel within the second range. A color difference generation process that is generated based on the second color component of each of the color pixels;
The second color difference is included in the second color pixel that is within the second range and is located above, below, left, and right of the first color pixel that is the target for generating the second color difference. A second color component, and a first color pixel that is in the second range and is located on the opposite side of the second color pixel from the first color pixel. Color difference generation processing to be generated based on the color components of
An image processing device that generates at least two second color differences of the first color pixel by performing at least two color difference generation processes. The image processing apparatus according to claim 1 .
The second color difference generation unit includes:
The second color difference of the first color pixel is within the second range with the first color component of the first color pixel that is the target for generating the second color difference. A color difference generation process that is generated on the basis of the second color component of each of the second color pixels adjacent above and below the first color pixel;
The second color difference of the first color pixel is within the second range with the first color component of the first color pixel that is the target for generating the second color difference. Color difference generation processing that is generated based on the second color components of the second color pixels adjacent to the left and right of the first color pixels,
The second color difference of the first color pixel is within the second range with the first color component of the first color pixel that is the target for generating the second color difference. A color difference generation process that is generated based on the second color component of each of the second color pixels adjacent to the upper and right sides of the first color pixel;
The second color difference of the first color pixel is within the second range with the first color component of the first color pixel that is the target for generating the second color difference. A color difference generation process that is generated based on the second color component of each of the second color pixels adjacent to the left and above the first color pixel;
The second color difference of the first color pixel is within the second range with the first color component of the first color pixel that is the target for generating the second color difference. A color difference generation process that is generated based on the second color component of each of the second color pixels adjacent to the lower and right sides of the first color pixel;
The second color difference of the first color pixel is within the second range with the first color component of the first color pixel that is the target for generating the second color difference. A color difference generation process that is generated based on the second color component of each of the second color pixels adjacent to the left and below the first color pixel;
The second color difference of the first color pixel is within the second range, and is adjacent to the top, bottom, left, and right of the first color pixel that is the target for generating the second color difference. The second color component of a second color pixel; and the first color pixel within the second range, adjacent to the second color pixel and relative to the second color pixel. a color difference generating process for generating on the basis of a first color component having the other of said first color pixels adjacent to the side opposite to the,
Of at least by performing two color difference generation processing, the first image picture processor that generates at least two of said second color difference with color pixels. The image processing apparatus according to claim 1 or 2 ,
The image generation unit interpolates the second color component at a first color pixel position of the first color pixel based on the second color component of the second color pixel, and Interpolating the third color difference at the second color pixel position of the second color pixel based on the third color difference of one color pixel, and the second color of the second color pixel Component, the third color difference of the first color pixel, the second color component at the first color pixel position, and the third color difference at the second color pixel position, images processing device that generates the output image. In the image processing device according to any one of claims 1 to 3 ,
When the other first color pixel located in the first range has the third color difference, the first color difference generator generates the other first color located in the first range. by performing the smoothing using the third color difference with pixels, images processing device that generates the first color difference. An image processing apparatus according to any one of claims 1 to 4 ,
It receives the object light, a camera and an image pickup device for generating the image data according to the respective spectral sensitivity of the first color component and said second color component included in the object light. The first color pixel having the spectral sensitivity of the first color component and the second color pixel having the spectral sensitivity of the second color component different from the first color component are output from the imaging device arranged. An image input unit for acquiring the obtained image data;
  A first color difference generation unit that generates the first color difference of the first color pixel by performing smoothing in a first range including the first color pixel;
  The second color difference of the first color pixel is included in the second color pixel of the second color pixel that is located in a second range that is smaller than the first range and that includes the first color pixel. A second color difference generation unit that generates based on color components;
  An image generation unit that generates an output image based on the color difference of the first color pixel based on the distribution range of the second color difference value and the second color component of the second color pixel; With
  The second color difference generation unit includes:
  The first color component of the first color pixel that is the target for generating the second color difference, the second color difference, and the first color pixel within the second range. A color difference generation process that is generated based on the second color components of the second color pixels positioned above and below,
  The first color component of the first color pixel that is the target for generating the second color difference, the second color difference, and the first color pixel within the second range. A color difference generation process that is generated based on the second color components of the second color pixels located on the left and right, respectively,
  The first color component of the first color pixel that is the target for generating the second color difference, the second color difference, and the first color pixel within the second range. One second color pixel of the second color pixel located above and below and the second one of the second color pixels located on the left and right of the first color pixel within the second range. A color difference generation process that is generated based on the second color component of each of the color pixels;
  The second color difference is included in the second color pixel that is within the second range and is located above, below, left, and right of the first color pixel that is the target for generating the second color difference. A second color component, and a first color pixel that is in the second range and is located on the opposite side of the second color pixel from the first color pixel. Color difference generation processing to be generated based on the color components of
  An image processing device that generates at least two second color differences of the first color pixel by performing at least two color difference generation processes. The first color pixel having the spectral sensitivity of the first color component and the second color pixel having the spectral sensitivity of the second color component different from the first color component are output from the imaging device arranged. image input processing and for obtaining the image data,
A first color difference generation process for generating the first color difference of the first color pixel by performing smoothing in a first range including the first color pixel;
The second color difference of the first color pixel is included in the second color pixel of the second color pixel that is located in a second range that is smaller than the first range and that includes the first color pixel. A second color difference generation process generated based on the color component;
The first color difference value is included when the second color difference value distribution range includes the first color difference value, and the distribution range when the first color difference value is not included in the distribution range. A color difference smoothing control process for generating a boundary value of the third color difference of the first color pixel;
Causing the computer to execute image generation processing for generating an output image based on the second color component of the second color pixel and the third color difference of the first color pixel ;
The second color difference generation process includes:
The first color component of the first color pixel that is the target for generating the second color difference, the second color difference, and the first color pixel within the second range. A color difference generation process that is generated based on the second color components of the second color pixels positioned above and below,
The first color component of the first color pixel that is the target for generating the second color difference, the second color difference, and the first color pixel within the second range. A color difference generation process that is generated based on the second color components of the second color pixels located on the left and right, respectively,
The first color component of the first color pixel that is the target for generating the second color difference, the second color difference, and the first color pixel within the second range. One second color pixel of the second color pixel located above and below and the second one of the second color pixels located on the left and right of the first color pixel within the second range. A color difference generation process that is generated based on the second color component of each of the color pixels;
The second color difference is included in the second color pixel that is within the second range and is located above, below, left, and right of the first color pixel that is the target for generating the second color difference. A second color component, and a first color pixel that is in the second range and is located on the opposite side of the second color pixel from the first color pixel. Color difference generation processing to be generated based on the color components of
An image processing program that generates at least two second color differences of the first color pixel by performing at least two color difference generation processes. The first color pixel having the spectral sensitivity of the first color component and the second color pixel having the spectral sensitivity of the second color component different from the first color component are output from the imaging device arranged. Image input processing to obtain the acquired image data;
  A first color difference generation process for generating the first color difference of the first color pixel by performing smoothing in a first range including the first color pixel;
  The second color difference of the first color pixel is included in the second color pixel of the second color pixel that is located in a second range that is smaller than the first range and that includes the first color pixel. A second color difference generation process generated based on the color component;
  An image generation process for generating an output image based on the color difference of the first color pixel based on the distribution range of the second color difference value and the second color component of the second color pixel; , Run on a computer,
  The second color difference generation process includes:
  The first color component of the first color pixel that is the target for generating the second color difference, the second color difference, and the first color pixel within the second range. A color difference generation process that is generated based on the second color components of the second color pixels positioned above and below,
  The first color component of the first color pixel that is the target for generating the second color difference, the second color difference, and the first color pixel within the second range. A color difference generation process that is generated based on the second color components of the second color pixels located on the left and right, respectively,
  The first color component of the first color pixel that is the target for generating the second color difference, the second color difference, and the first color pixel within the second range. One second color pixel of the second color pixel located above and below and the second one of the second color pixels located on the left and right of the first color pixel within the second range. A color difference generation process that is generated based on the second color component of each of the color pixels;
  The second color difference is included in the second color pixel that is within the second range and is located above, below, left, and right of the first color pixel that is the target for generating the second color difference. A second color component, and a first color pixel that is in the second range and is located on the opposite side of the second color pixel from the first color pixel. Color difference generation processing to be generated based on the color components of
  An image processing program that generates at least two second color differences of the first color pixel by performing at least two color difference generation processes.

Images