JP5268321B2 - Image processing apparatus, image processing method, and image processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To prevent zipper artifact and jaggy generated along an image boundary of red and blue to obtain a high-definition color image, when the color image is formed from a mosaic image output from a single CCD imaging device of a Bayer constellation. <P>SOLUTION: As S300 (brightness formation step), in S310, a color mosaic image is processed at a low pass filter to remove a color carrier and to generate a first brightness value for each pixel. In S320, a G pixel is removed from the color mosaic image to form a BR plane composed of a B pixel and an R pixel. In S330, the BR plane is processed at a bandpass filter to calculate an brightness correction value. In S340, the correction value formed at S330 is added to the first brightness value to generate a second brightness value. <P>COPYRIGHT: (C)2009,JPO&amp;INPIT

Description

  The present invention relates to an image processing apparatus and image processing for generating a color image from a color mosaic image output via an image pickup device having a color filter with a Bayer array of three colors of R (red), G (green), and B (blue). More particularly, the present invention relates to a technique for generating a luminance signal from an output signal output via an image sensor.

  2. Description of the Related Art Conventionally, in an imaging system such as a digital camera, an image processing apparatus and an image processing method are known in which a subject image is formed on an image sensor via a lens, the subject image is photoelectrically converted by the image sensor, and an image signal is generated. It has been.

  As a single-plate image sensor, a plurality of photoelectric conversion elements are provided in a matrix, and a color filter is provided in front of the image sensor, and image processing is performed by applying signal processing to the pixel signals of each color output via the color filter. An image processing apparatus and an image processing method to be generated are known.

  In a so-called single-plate image sensor using only one image sensor, specific color filters corresponding to each pixel on the image sensor, such as red (R), green (G), and blue (B) colors, are used. A filter is provided. Usually, a pixel provided with an R color filter is an R pixel (red pixel), a pixel provided with a G color filter is G pixel (green pixel), and a pixel provided with a B color filter is B pixel (blue pixel). Pixel).

  A color filter called a primary color Bayer array is used as a representative color filter in a single-plate image sensor. The color filter of the primary color Bayer arrangement is such that rows in which R and G are alternately arranged in the horizontal direction and rows in which G and B are alternately arranged in the horizontal direction are alternately arranged in the vertical direction. A plurality of basic blocks each having two vertical pixels (2 × 2 pixels) as one unit (one unit) are periodically arranged. Each basic block has a structure in which two green pixels are arranged on one diagonal and a red pixel and a blue pixel are arranged on the other diagonal.

  In some image processing apparatuses, a luminance signal and a color difference signal are generated from an image signal output via a single-plate image sensor, and an image signal of a color image is generated from these signals (for example, Patent Document 1).

  Specifically, in a single-plate image sensor, each pixel has only single color information, but in order to display a color image, all the values of red (R), green (G), and blue (B) are each pixel. Is necessary. For this reason, in image processing using a single-plate image sensor, so-called demosaic processing is performed based on a color mosaic image in which each pixel has only one of R, G, and B components. Here, the demosaic process is an interpolation operation using the luminance information of other insufficient colors collected from the surrounding pixels for the single color information of each pixel of the color mosaic image, so that each pixel is R, This is processing for generating a color image having all of the G and B components (so-called color interpolation processing).

In addition, as a simple method for generating a luminance signal, a color carrier component is obtained by directly filtering a color mosaic image output from an image sensor using, for example, a low-pass filter represented by (Equation 1). There is known a method of removing luminance and generating luminance using a pixel signal obtained through a filter (see, for example, Patent Document 2). That is, a pixel signal output through a uniform low-pass filter for a color mosaic image is used as a luminance signal (hereinafter referred to as SwitchY luminance generation method).
JP 2000-341702 A JP 2000-287219 A

  However, according to the conventional method of generating luminance by applying a low-pass filter to a predetermined pixel range centering on the pixel of interest, light and darkness is jagged, particularly along the image boundary between an oblique red image and a blue image. Pattern zipper artifacts occurred and there was room for further improvement. Here, the zipper artifact refers to a phenomenon in which light and darkness that does not exist in the original image appear in a zipper shape (a so-called slide fastener shape) along the image boundary by demosaic processing.

  Specifically, as shown in FIG. 9A, blue having a luminance of (R, G, B) = (0, 0, 1) in the upper right region (RU in the figure) via the image boundary L. When an image is formed and a red image having a luminance of (R, G, B) = (1, 0, 0) is formed in the lower left region (LD in the figure), the color mosaic image is shown in FIG. As shown in (b), B pixel is 1 in the image range in the upper right region, R pixel is 1 in the image range in the lower left region, and all other pixels are 0.

  Then, when the image signal shown in FIG. 9B is subjected to the low-pass filter of (Equation 1) to remove the color carrier component, as shown in FIG. The pixel values 3/16 and 5/16, which are undesirable, are alternately generated, and a light and dark pattern called a zipper artifact is generated. Further, such a luminance mismatch at the image boundary is not limited to the boundary between red and blue, but easily occurs in an area where the color difference between red and blue changes, and appears as jagged in the color image.

  Therefore, the present invention suppresses zipper artifacts and jaggies that occur along the image boundary between red and blue when generating a color image from a mosaic image output from a single-plate imaging device with a primary color Bayer array, An object of the present invention is to provide an image processing apparatus and an image processing method capable of obtaining a quality color image.

In order to achieve this object, the invention according to claim 1 is output via an image sensor having a Bayer array color filter of R (red), G (green), and B (blue). An image processing apparatus for generating a color image having a plurality of color information based on a color mosaic image, wherein a predetermined pixel range including a target point for luminance generation is set on the mosaic image, and the pixel Based on the R, G, and B pixel values located within the range, a luminance generation unit that generates luminance at the target pixel is provided, and the luminance generation unit applies a low-pass filter to the color mosaic image. First brightness generating means for removing a color carrier and generating a brightness value for each pixel, and a BR plane for generating a BR color mosaic image composed of B and R pixels by removing the G pixel from the color mosaic image A correction value generating means for filtering the BR plane by applying a band filter, and obtaining an output value for each pixel output through the band filter as a luminance correction value; and the first luminance Second luminance generation means for generating a second luminance value by adding the correction value generated by the correction value generation means to the first luminance value generated by the generation means, and the second luminance The value is a luminance value in the color image.

According to the image processing apparatus of claim 1 , the luminance generation unit applies a low-pass filter to the color mosaic image to remove the color carrier, and generates a luminance value for each pixel. And a BR plane generation unit that generates a BR color mosaic image including B and R pixels by removing G pixels from the color mosaic image, and filters the BR plane by applying a band filter. A correction value generating means for obtaining an output value for each pixel output as a luminance correction value, and adding the correction value generated by the correction value generating means to the luminance value generated by the first luminance generating means. And a second luminance generation means for generating a second luminance value, so that the zipper artifact appearing in the first luminance is canceled with the correction value, and the zipper artifact is suppressed. It can be obtained second luminance.

According to a first aspect of the present invention, as in the second aspect of the present invention, the low-pass filter is simply a convolution by a 3 × 3 pixel kernel. A first luminance can be generated.

The image processing apparatus according to claim 1, as in the invention according to claim 3, by a convolution (convolution) by the kernel (kernal) of the bandpass filter 5 × 5 pixels, conveniently A second luminance can be generated.

Next, the invention according to claim 4 is based on a color mosaic image output through an image sensor having a Bayer array color filter of R (red), G (green), and B (blue). An image processing method for generating a color image having a plurality of color information, wherein a predetermined pixel range including a target point for luminance generation is set on the mosaic image, and is located within the pixel range. Based on the R, G, and B pixel values, a luminance generation step for generating luminance at the target pixel is used. In the luminance generation step, a color carrier is removed by applying a low-pass filter to the color mosaic image, A first luminance generation step for generating a first luminance value for each pixel, and a BR plane generation step for generating a BR plane composed of B and R pixels by removing the G pixel from the color mosaic image. A correction value generating step of filtering the BR plane by applying a band filter, and obtaining an output value for each pixel output through the band filter as a luminance correction value; and the first luminance generation step A second luminance generation step of generating a second luminance value by adding the correction value generated in the correction value generation step to the first luminance value generated by The brightness value in the color image is used.

According to the image processing method of claim 4 , a predetermined pixel range including a target point for luminance generation is set on the mosaic image, and based on the R, G, and B pixel values located in the pixel range. Then, a luminance generation step for generating luminance at the target pixel is used, and in the luminance generation step, the color mosaic image is subjected to low-pass filtering to remove the color carrier, and the first luminance value for each pixel is generated. One luminance generation step, a BR plane generation step of generating a BR plane composed of B and R pixels by removing the G pixel from the color mosaic image, and filtering the BR plane by applying a bandpass filter, The correction value generation step for obtaining the output value for each pixel output through the bandpass filter as the luminance correction value, and the luminance generated by the first luminance generation step Using the second luminance generation step of generating a correction value second luminance value added generated correction value generating steps for, since the second luminance values and the luminance values in a color image, according to claim 1 As in the invention described in (1), the zipper artifact appearing in the first luminance can be canceled with the correction value, and the second luminance in which the zipper artifact is suppressed can be obtained.

The image processing method according to claim 4, as in the invention according to claim 5, by the low-pass filter is a 3 × 3 pixel kernel (kernal) by convolution (convolution), claim Similarly to the invention described in 2 , the first luminance can be easily generated.

Further, in the image processing method according to claim 4 or 5 , as in the invention according to claim 6 , the bandpass filter is a convolution with a kernel of 5 × 5 pixels. Thus, the second luminance can be easily generated as in the third aspect of the invention.

Next, the invention described in claim 7 is based on a color mosaic image output via an image pickup device having a Bayer array color filter of R (red), G (green), and B (blue). An image processing program for generating a color image having a plurality of color information, wherein a predetermined pixel range including a target point for luminance generation is set on the mosaic image, and is located within the pixel range. Based on the R, G, and B pixel values, a luminance generation step for generating luminance at the target pixel is used. In the luminance generation step, a color carrier is removed by applying a low-pass filter to the color mosaic image, A first luminance generation step for generating a first luminance value for each pixel; and a BR plane generation for generating a BR plane composed of B and R pixels by removing the G pixel from the color mosaic image. A correction value generating step of filtering the BR plane by applying a band filter and obtaining an output value for each pixel output through the band filter as a luminance correction value; and the first luminance The correction value generated in the correction value generation step is added to the first luminance value generated in the generation step to generate a second luminance value, and the second luminance value is used as the luminance value in the color image. And generating a second luminance generation step. The computer executes the second luminance generation step.

According to the image processing program of claim 7 , in the luminance generation step, a first low-pass filter is applied to the color mosaic image to remove the color carrier and a first luminance value for each pixel is generated. A luminance generation step, a BR plane generation step of generating a BR plane composed of B and R pixels by removing G pixels from the color mosaic image, and filtering by applying a band filter to the BR plane; The correction value generation step for obtaining the output value for each pixel output via the correction value as the luminance correction value, and the correction value generated in the correction value generation step with respect to the luminance value generated in the first luminance generation step In addition, a second luminance value is generated, and a second luminance generation step for generating the second luminance value as a luminance value in the color image is executed on the computer. By, as in the invention described in claim 1, the zipper artifact appearing in the first luminance canceled by the correction value, the zipper artifact can be obtained second luminance was suppressed.

The image processing program according to claim 7, as in the invention according to claim 8, by the low-pass filter is a 3 × 3 pixel kernel (kernal) by convolution (convolution), claim Similarly to the invention described in 2 , the first luminance can be easily generated.

Further, in the image processing program according to claim 7 or claim 8 , as in the invention according to claim 9 , the bandpass filter is a convolution based on a kernel of 5 × 5 pixels. Thus, the second luminance can be easily generated as in the third aspect of the invention.

The image processing apparatus, the image processing method, and the image processing program of the present invention apply a low-pass filter to a color mosaic image to remove a color carrier and generate a first luminance value for each pixel when generating luminance. And generating a BR plane composed of B and R pixels by removing the G pixel from the color mosaic image, filtering the BR plane by applying a band filter, and outputting the result through the band filter. The output value for each pixel is obtained as a luminance correction value, and the second luminance value is generated by adding the correction value to the first luminance value, thereby canceling the zipper artifact appearing in the first luminance with the correction value. Thus, the second luminance in which the zipper artifact is suppressed can be obtained.

(First embodiment)
Next, an embodiment of an image processing apparatus, an image processing method, and an image processing program according to the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram showing the configuration of an imaging apparatus according to the first embodiment to which the present invention is applied, FIG. 2 is a diagram showing the configuration of a color mosaic image with a Bayer arrangement in the same embodiment, and FIG. a diagram of a luminance generation in the form, (a) represents a luminance Y ₀ tentative generated by SwitchY generator represents the correction value D (b) is generated so as to correct the luminance Y _0, (c) is a diagram showing a luminance Y ₁ that is corrected by subtracting a predetermined amount of the correction value D from the luminance Y ₀ tentative. FIG. 4 is a flowchart showing the procedure of the image processing method and the image processing program in the embodiment.

  As shown in FIG. 1, the imaging device 1 outputs the subject image to the imaging device 5 and outputs it as a digital image signal C (which is a mosaic image signal), and is output via the imaging unit 2. The image processing apparatus 100 generates a color image having a plurality of color information for each pixel based on the digital image signal C.

  The imaging unit 2 is output from the imaging lens 3 that guides the subject image P to the imaging device 5, the imaging device 5 that converts the received imaging light into an electrical quantity and outputs the same (for example, CCD: Charge Coupled Devices) 5, and the imaging device 5. AFE (Analog Front End) 6 that converts the analog image signal to be converted into a digital image signal C and outputs it, TG (Timing Generator) 13 that controls the image sensor 5 and AFE 6 at a predetermined cycle, and the optical axis direction of the imaging lens 3 ( A lens driving unit 12 that performs slide driving in the Z direction in FIG. 3, a detection unit 10 that detects the sliding amount of the imaging lens 3 in the Z direction via the sensor 11, and the like are provided. Note that the present invention is not limited to the presence or absence of the imaging lens 3, the lens driving unit 12, the sensor 11, the detection unit 10, and the like, and as will be described later, the imaging device 5 having a color filter with an RGB three-color Bayer array. This relates to the image processing of the color mosaic image output via.

  The imaging element 5 is configured by arranging a plurality of photoelectric conversion elements in a matrix, and is configured to photoelectrically convert an imaging signal for each photoelectric conversion element and output an analog image signal.

  The image sensor 5 includes a color filter 5a having a Bayer array of three colors R (red), G (green), and B (blue) in association with the photoelectric conversion elements, and the amount of light that has passed through the filter portions of the respective colors. Is converted into an electrical signal.

  The AFE 6 is a correlated double sampling circuit (CDS: Correlated Double Sampling) 7 that removes noise from the analog image signal output via the image sensor 5, and an image that has been correlated double sampled by the correlated double sampling circuit 7. A variable gain amplifier (AGC) 8 that amplifies the signal, an A / D converter 9 that converts an analog image signal from the image sensor 5 input through the variable gain amplifier 9 into a digital image signal, and the like The image signal configured and output from the image sensor 5 is converted into a digital image signal C in association with the photoelectric conversion element and output to the image processing apparatus 100.

  In the imaging unit 2, a CMOS (Complementary Metal Oxide Semiconductor) sensor may be used instead of the imaging device 5, the correlated double sampling circuit 7, the variable gain amplifier 8, the A / D converter 9, and the like. Since the signal for each pixel output from the image pickup device 5 has only single color information, a mosaic image signal is output from the image pickup unit 2 to the image processing apparatus 100.

  Next, the image processing apparatus 100 separates the mosaic image output from the imaging unit 2 for each of R, Gr, Gb, and B pixels, and the pixel signal output from the color plane decomposition unit 21 A luminance generation unit 25 for generating luminance as a color image signal at a pixel position of the color image based on the color difference and a color difference generation unit 30 for generating color difference are provided.

  The image processing apparatus 100 also performs a known gamma correction, saturation correction, and edge enhancement for improving the appearance of the color image with respect to the color image signals output from the luminance generation unit 25 and the color difference generation unit 30. The color image output through the correction unit 34 and the visual correction unit 34 is compressed by a method such as JPEG, and the color image output through the compression unit 35 is stored in a recording medium such as a flash memory. Recording unit 36, CPU (Central Processing Unit) 18, ROM (Red Only Memory) 19 and the like. The CPU 18 performs each process of the image processing apparatus 100 and the imaging apparatus 1 according to a control program stored in the ROM 19. Control.

  The color plane separation unit 21 is associated with the Bayer array illustrated in FIG. 2 and stores an R field memory 22 that stores R pixel signals and a pixel signal of Gr (R adjacent to R in one direction). A Gr field memory 23a, a Gb field memory 23b that stores a pixel signal of Gb (R adjacent to B in one direction), and a B field memory 24 that stores a pixel signal of B. Based on the above, these pixel signals are output to the luminance generation unit 25 and the color difference generation unit 30. Note that each color field memory is a virtual object, and in actuality, each color field may be assigned by address so that each color field can be distinguished on a single memory.

  Next, the luminance generation unit 25 sets a pixel range including the target point (target pixel) for each of R, G, and B in the arrangement of the color mosaic images, and sets the R, G, and B positions in this pixel range. The luminance of the target point is generated based on the pixel value.

Luminance generation section 25, SwitchY generator 26 for generating a luminance Y ₀ provisional sum multiplied by a predetermined coefficient to the pixel value of the pixels surrounding the target pixel and the target pixel, the color mosaic image except for pixels of G An RB plane generation unit 27 that generates an RB plane consisting only of R and B pixel values, a luminance correction value generation unit 28 that generates a luminance correction value D by applying a predetermined bandpass filter to the RB plane, and provisional luminance A luminance correction unit 29 that generates a second luminance Y ₁ by subtracting an amount obtained by multiplying the correction value D by a predetermined coefficient from Y ₀ is configured. Note that its function by the luminance generation unit luminance generation unit 25 in the first aspect of the present invention is expressed, the first luminance generation unit in the first aspect of the present invention is expressed its function by SwitchY generating unit 26, the present invention The function of the second luminance generating means according to claim 1 is expressed by the luminance correcting unit 29.

Specifically, in the SwitchY generation unit 26, each pixel value located in the 3 × 3 pixel range centered on the pixel of interest in the color mosaic image is multiplied by the low-pass filter expressed by (Equation 1), and the pixel values are totaled. luminance Y ₀ provisional of the pixel of interest is calculated. That is, three pixels in the uv direction centered on the target pixel are set and weighted according to each pixel position, and the sum of these is set as the temporary luminance Y ₀ of the target pixel. In this embodiment, as shown in FIG. 9A, the color mosaic image has a luminance of (R, G, B) = (0, 0, 1) in the upper right region via the image boundary L. And a blue and red image are formed so as to have a luminance of (R, G, B) = (1, 0, 0) in the lower left region, and will be described below. .

In SwitchY generator 26, the luminance _{Y 0} provisional as shown in FIGS. 3 (a) is determined. In FIG. 3A, 1/16 of the pixel value entered at each pixel position is a provisional luminance Y ₀ (so-called first luminance in the present invention). For example, the luminance Y ₀ of the pixel A ₁₁ located at (u, v) = (1, 1) in FIG. 3A is Y ₀ = (A ₀₀ × 1 + A ₁₀ × 2 + A ₂₀ × 1 + A ₀₁ × 2 + A ₁₁ × _{4 + a 21 × 2 + a} 02 × 1 + a 12 × 2 + a 22 × 1) / 16 is calculated by the calculation equation, _{a 00} = ₀ on the calculation equation with reference to FIG. 9 (b), a 10 = 0, a 20 = 0 Substituting A ₀₁ = 0, A ₁₁ = 0, A ₂₁ = 1, A ₀₂ = 0, A ₁₂ = 1, and A ₂₂ = 0, Y ₀ = 4/16 is obtained.

  In the RB plane generation unit 27, since the pixel value located at the G pixel in the color mosaic image is 0, the B pixel in the upper right region and the lower left region are passed through the image boundary L as in FIG. 9B. An RB plane having a pixel value at the position of the R pixel in the region is generated.

Next, in the luminance correction value generation unit 28, each pixel value located within the 5 × 5 pixel range with the target pixel as the center in the RB plane generated by the RB plane generation unit 27 is expressed by (Expression 2). The pixel values are summed by applying a bandpass filter, and the luminance correction value D at the target pixel is obtained. That is, five pixels in the uv direction centered on the target pixel are set and weighted according to the pixel position, and the sum of these is set as the correction value D of the target pixel.

Then, as shown in FIG. 3B, the correction value D is generated for the B pixel and the R pixel. In FIG. 3B, 1/64 of the numerical value entered at each pixel position is the correction value D. For example, the correction value D of the pixel A ₃₂ located at (u, v) = (3, 2) in FIG. 3A is D = (A ₁₀ × (−1) + A ₂₀ × (−2) + A ₃₀ × (−2) + A ₄₀ × (−2) + A ₅₀ × (−1) + A ₁₁ × (−2) + A ₂₁ × (0) + A ₃₁ × (4) + A ₄₁ × (0) + A ₅₁ × (−2 ) + A ₁₂ × (−2) + A ₂₂ × (4) + A ₃₂ × (12) + A ₄₂ × (4) + A ₅₂ × (−2) + A ₁₃ × (−2) + A ₂₃ × (0) + A ₃₃ × ( 4) + A ₄₃ × (0) + A ₅₃ × (−2) + A ₁₄ × (−1) + A ₂₄ × (2) + A ₃₄ × (2) + A ₄₄ × (−2) + A ₅₄ × (−1)) / 64, and the pixel values at the pixel positions A ₂₁ , A ₄₁ , A ₁₂ , A ₄₃ , A ₁₄ , and A ₃₄ are set to 1 with reference to FIG. 9B. If the pixel values at other pixel positions are substituted as 0, a numerical value (−5) / 64 is obtained.

At this time, the bandpass filter expressed by (Expression 2) may be applied by being divided into two stages of filters as expressed by (Expression 3).

Next, the brightness correcting unit 29, by subtracting the amount obtained by multiplying the coefficient k = 2/3 in the correction value from the provisional luminance _{Y 0} generated by SwitchY generator 26, as shown in FIG. 3 (c) second obtaining luminance _{Y 1.}

In FIG. 3 (c), 1/96 of numbers entered in each pixel position is the second luminance Y _1. Here, the provisional luminance Y ₀ (FIG. 3A) generated by the SwitchY generation unit 26 and the second luminance Y ₁ corrected by the luminance correction unit 29 (FIG. 3C) are used. By comparison, it can be seen that along the image boundary L, the luminance change of the luminance Y ₁ is less than the temporary luminance Y ₀ and the zipper artifact is suppressed.

  Regarding the coefficient k, k = 2/3 is optimal for suppressing zipper artifacts, but if the range is 0 <k <(4/3), the effect of suppressing zipper artifacts can be obtained. Since the numerical value 2/3 is expressed as 0.101010 ... b in binary number, k is in the range of 1/2 (0.1b in binary number) to 3/4 (0.11b in binary number). In this case, the binary number may be set to a simple numerical value.

Further, the luminance generation unit 25 obtains the correction value D for the RB plane using (Equation 2), and subtracts the amount obtained by multiplying the correction value D by a coefficient from the temporary luminance Y ₀ to obtain the second luminance Y. was calculated _1, these alternative, for the pixel position of the G color mosaic calculates a second luminance Y ₁ using (equation 4), to the pixel position of the B and R ( equation 5) second may calculate the luminance Y ₁ using. As a result, the second luminance Y ₁ calculated by the second luminance Y ₁ and calculated by the luminance correction section 29 (Equation 4) and (5) have the same value.

In (Expression 4) and (Expression 5), the 5 × 5 matrix on the left side is obtained by setting the value located at the G pixel to 0, and if this 0 pixel is ignored, the pixel of interest is the center. Four or five G pixels present in the peripheral 3 × 3 pixels and four or nine R and B pixels present in the peripheral 5 × 5 pixels are used for calculating the second luminance Y ₁ . .

Also, when calculating the second brightness Y _1, the G plane obtained by extracting only G pixels multiplied by the filter (Equation 1), whereas, for the RB plane obtained by extracting only the R and B pixel (formula multiplied by the filter 6), may be the sum of the pixel values output through both filters as the second luminance Y _1.

Further, the filter of (Expression 6) may be configured by a two-stage filter as expressed in (Expression 7).

  Next, the color difference generation unit 30 generates a color difference C1 and a color difference C2 that are color components of the target pixel using each pixel value included in a predetermined pixel range centered on the target pixel. Here, C1 represents a value obtained by subtracting the R pixel signal and the B pixel signal from the sum of the Gr pixel signal and the Gb pixel signal, while C2 represents the R pixel signal and the B pixel signal. Represents the difference between Specifically, in a predetermined pixel range, C1 is generated so that the composition ratio of R: G: B = (− 1): 2: (− 1), while R: G: B = 2: C2 is generated so that the composition ratio is 0: (− 2).

An example of generation of C1 and C2 is described below. First, in the pixel range centered on the target pixel, the filter of (Equation 8) is applied to the color mosaic image when the target pixel is the G pixel, and to the color mosaic image when the target image is the B and R pixels ( The filter of Expression 9) is applied, and the result is set as the color difference C1 at the target pixel position.

Further, in the pixel range centered on the target pixel, the filter of (Equation 10) is applied to the color mosaic image for the R pixel, and the filter of (Equation 11) is applied to the color mosaic image for the Gr pixel. For the Gb pixel, the filter of (Equation 12) is applied to the color mosaic image, and for the B pixel, the filter of (Equation 13) is applied to the color mosaic image, and the result is the color difference C2 at the target pixel position. Note that the present invention is an invention relating to luminance generation, and is not limited to generation of color differences.

Next, vision correcting unit 34 adds a predetermined processing on the luminance signal Y ₁ and the color difference signals C1, C2 obtained by the color difference generation unit obtained by the luminance signal generating unit 25, consisting of the color signals Correct the appearance of the color image. For example, reducing the false color or color noise by applying a low-frequency filtering on the color difference signals C1, C2, or emphasize a sense of resolution by performing high frequency enhancement on the luminance signal Y _1.

Note that the luminance signal Y ₁ generated by the embodiment of the present invention has a modulation factor near fs / 4 (fs is a sampling frequency) in a black and white subject, as compared with the luminance signal Y ₀ generated by the SwitchY method. Therefore, it is desirable to perform high frequency emphasis so as to increase the degree of modulation near fs / 4 via the visual correction unit 34.

Further, when the compression unit 35 compresses the image signal, the visual correction unit 34 uses the YUV chromaticity coordinates as the color space coordinates of the standard color image, so the luminance signal Y ₀ generated by the luminance generation unit 25 is used. The color differences C1 and C2 generated by the color difference generation unit 35 are converted into YUV values (Y is the luminance of the pixel, U is the color difference between the luminance Y and blue (B), and V is the luminance Y and red (R)). Color difference). For more conversion, because of the relationship between the luminance signal Y ₀ and color difference C1 generated by the color difference generation unit 30, C2 which is generated by the luminance generation section 25 (Equation 14), the YUV values using equation (15) I do.

  Next, the procedure of the image processing method and the image processing program of the first embodiment will be described with reference to FIG. This procedure is executed by the CPU 18 giving a command signal to each functional unit based on a program stored in the ROM 19. Further, S in FIG. 4 represents a step.

  First, this procedure starts when an activation signal is input to the imaging apparatus 1 or the image processing apparatus 100 by an operator.

  Next, in S100, an image signal (color mosaic image) is read into the image processing apparatus 100 via the imaging unit 2, and the R plane signal, the Gr pixel signal, and the Gb pixel are associated with the Bayer array by the color plane decomposition unit 21. Each signal and B pixel signal is stored, and then the process proceeds to S200.

  Next, in S200, a target point (target pixel) is set corresponding to the pixel arrangement of the color mosaic image, and then the process proceeds to S300. At this time, the target point to be set is any pixel of the color mosaic image, and is determined by a program stored in the ROM 19.

Next, in S300, the luminance of the target point (target pixel) is generated. Specifically, as illustrated in FIG. 4B, in S310, the SwitchY generation unit 26 is used, and each pixel value positioned in a 3 × 3 matrix centered on the pixel of interest in the color mosaic image is expressed by (Expression 1). by applying a low-pass filter that represents the sum of the pixel values to obtain the luminance Y ₀ of the temporary target pixel, then it goes to S340.

  On the other hand, in S320, the RB plane generating unit 27 is used to generate an RB plane in which the pixel value located at the G pixel in the color mosaic image is 0 and only the R and B pixels have pixel values, and then the process proceeds to S330.

  Next, in S330, the brightness correction value generation unit 28 is used, and in the RB plane generated by the RB plane generation unit 27, each pixel value located in a 5 × 5 matrix with the target pixel as the center is expressed by (Expression 2). The correction value D at the target pixel is generated by applying the bandpass filter, and then the process proceeds to S340.

Then, in S340, using the luminance correction unit 29 obtains a luminance _{Y 1} by subtracting the amount obtained by multiplying a predetermined coefficient k from the luminance _{Y 0} tentative generated in the correction value D generated at S330 in S310, then, FIG. The process proceeds to S500 of 4 (a).

  Next, in S500, it is determined whether or not there is a next target point (target pixel) for generating luminance. When there is a next target point (Yes), S200 to 500 are repeated, and the next target point ( When there is no (pixel of interest) (No), the process proceeds to S600.

Then, in S600, using a vision correcting unit 34, the color image signal generated by the luminance generation section 25 and a color difference generating section 30 (luminance Y _1, chrominance C1, chrominance C2) with respect to, for improving the appearance of the image Image correction, conversion to YUV chromaticity coordinates, which are color space coordinates of a standard color image, and the like, and then the process proceeds to S700.

  Next, in S700, the digital image signal of the color image output via the visual correction unit 34 is compressed by a method such as JPEG (Joint Photographic Experts Group) using the compression unit 35, and the size of the image data at the time of recording is recorded. After that, the process proceeds to S800.

  In step S800, the compressed digital image signal is stored in a recording medium such as a flash memory using the recording unit 36, and then the image processing program ends.

As described above, according to the image processing apparatus 100 and the image processing method and image processing program according to the first embodiment, the color mosaic image, and generates a luminance Y ₀ provisional using SwitchY generator 26 , using the RB plane generating unit 27 and the luminance correction value generation unit 28 generates the correction value D, then, using the luminance correction unit 29 subtracts the correction value from the luminance Y _0, the luminance Y which suppresses zipper artifact ₁ can be generated, and a high-quality color image can be obtained.

(Second Embodiment)
Next, a second embodiment of the present invention will be described with reference to FIGS. FIG. 5 is a block diagram showing the configuration of the imaging apparatus 1A according to an embodiment to which the image processing apparatus, the image processing method, and the image processing program of the present invention are applied.

  FIG. 6 is an explanatory diagram for generating the B @ R plane and the R @ B plane in the second embodiment, FIG. 7 is an explanatory diagram for sampling in the color generation in the second embodiment, and FIG. It is a flowchart showing the procedure of the image processing method and image processing program in 2nd Embodiment.

  Note that the imaging apparatus 1A in the second embodiment is basically the same in configuration as the imaging apparatus 1 shown in the first embodiment, and therefore, the same components are assigned the same reference numerals and detailed description is given. The following is a description of the characteristic features.

  In the imaging unit 2A according to the present invention, the imaging lens 3 is configured to be slidable in the optical axis direction via the lens driving unit 12, and the lens state such as the focal length of the imaging lens 3 and the subject distance from the imaging lens 3 to the subject is set. It is configured to be variable.

  As shown in FIG. 5, an angular velocity sensor (for example, a gyroscope) 15 that detects a shake of the image pickup apparatus 1 </ b> A and outputs an electrical signal (hereinafter referred to as a shake signal) corresponding to the shake amount of the image pickup unit 2 </ b> A. The image processing apparatus 100A is provided with a blur detection unit 40 that detects the blur amount of the imaging apparatus 1A via the angular velocity sensor 15.

  The image processing apparatus 100A also separates the input image (mosaic image) output from the imaging unit 2A for each of R, Gr, Gb, and B pixels, stores the color plane decomposition unit 21, the focal length, and the subject. A lens state detection unit 37 for detecting the lens state of the imaging lens 3 such as a distance, an aberration coefficient storage table 38 storing a distortion aberration coefficient in association with the lens state of the imaging lens 3, a lens state detection unit 37, and an aberration coefficient recording unit. An aberration coefficient setting unit 39 that sets parameters for distortion aberration based on information from 38, blur correction based on detection values from the blur detection unit 40, and aberration correction based on setting values from the aberration coefficient setting unit 39 are performed. A sampling coordinate setting unit 41 for setting a sampling position in the input image is provided.

  Further, the image processing apparatus 100A replaces the interpolation value of the R pixel surrounding the B pixel with the B pixel value in the B pixel arrangement, and sets the R pixel value @B pixel position (hereinafter referred to as R @ B). R @ B plane generation unit 51 for generating the interpolation, the interpolation value of the B pixel surrounding the R pixel is replaced with the R pixel value at the R pixel position, and the B pixel value @R pixel position (hereinafter referred to as B @ R B @ R plane generation unit 52 for generating interpolation as a pixel value at the sampling position of the mosaic image for each of the R plane, Gr plane, Gb plane, B plane, R @ B plane, and B @ R plane. Sampling unit 53 that generates sampling values for each plane, and R plane, Gr plane, Gb plane, B plane, R @ B plane, and B @ R plane generated by sampling unit 53 And a color generation unit 54 generates the color information having a plurality of color component sampling values combined and for each pixel, such as.

  As shown in FIG. 7, the color plane separation unit 21 extracts an R plane (b) in which only R pixels are extracted from a color mosaic image in the Bayer array in (a), and a Gr plane (c) in which only Gr pixels are extracted. , Gb plane (d) from which only Gb pixels are extracted, and B plane (e) from which only B pixels are extracted. At this time, the G pixel arranged in the horizontal direction with respect to the R pixel is designated as Gr, and the G pixel arranged in the horizontal direction with respect to the B pixel is designated as Gb. Further, here, as cited in the first embodiment, the color mosaic image has a luminance of (R, G, B) = (0, 0, 1) in the upper right region via the image boundary L. Suppose that blue and red images are defined and formed so as to have a luminance of (R, G, B) = (1, 0, 0) in the lower left region.

Next, as shown in FIGS. 6A to 6C, the B @ R plane generating unit 51 generates the value of the B pixel at the position of the R pixel in the color mosaic image of FIG. Specifically, the B @ R plane is a luminance signal obtained by re-sampling the B plane at the R pixel position, and is generated by applying the filter of (Equation 16) to the B plane.

For _example, the luminance signal of the B @ R plane at the pixel position of _{A 32} B _uses the pixel values of _{A 21, A 41, A 23} , A 43 around the pixel of the _{A 32, (A 21 × 1} + A 41 × 1 + A ₂₃ × 1 + A ₄₃ × 1) / 4, and the pixel values of A ₂₁ , A ₄₁ , and A ₄₃ are set to 1, and the pixel of A ₂₃ is calculated with reference to FIG. 6B. Substituting the value as 0 gives the luminance signal 3/4.

  As a result, as shown in FIG. 6C, the B @ R plane has a value at the R pixel position, and a value of 0 at other pixel positions excluding the R pixel position.

  Next, as illustrated in FIGS. 6A, 6D, and 6E, the R @ B plane generation unit 52 calculates the value of the R pixel at the position of the B pixel in the color mosaic image in FIG. Generate. Specifically, the R @ B plane is a luminance signal obtained by re-sampling the R plane at the position of the B pixel, and is generated by applying the filter of (Expression 16) to the R plane.

For _example, the luminance signal of R of R @ B _plane, using the pixel values of _{A 32, A 52, A 34} , A 54 around the pixel of the _{A 43} at the pixel position of _{A 43, (A 32 × 1} + A 52 × 1 + A ₃₄ × 1 + A ₅₄ × 1) / 4, and the pixel values of A ₃₂ , A ₅₂ , and A ₅₄ are set to 0 and pixels A ₃₄ with reference to FIG. Substituting the value as 1, the luminance signal 1/4 is obtained.

  As a result, as shown in FIG. 6E, the R @ B plane has a value at the B pixel position, and a value of 0 at other pixel positions excluding the B pixel.

  Next, the sampling coordinate setting unit 41 considers image deformation such as distortion correction, camera shake correction, and digital zoom, and outputs the pixel of the color image to be output based on the parameters acquired by the aberration coefficient setting unit 39 and the shake detection unit 40. Set sampling coordinates on the color mosaic image corresponding to the position.

Here, it is assumed that the pixel position represented by (Expression 17) in the color mosaic image is transformed into the pixel position on the output image represented by (Expression 18). In (Equation 18), (u _s , v _s ) represents the coordinates of the pixel position of the output image, and f represents a deformation function for image deformation.

For example, the pixel position of the color image to be output is (Equation 19) if the zoom magnification is Z times (Equation 19), and is the image deformation of the rotation angle θ (Equation 20). At this time, the sampling coordinate setting unit 41 calculates the sampling coordinates (formula 22) on the mosaic image before the transformation for the output pixel represented by (formula 21). The pixel position is set on an integer grid in the (u, v) coordinate system, but the value of the sampling coordinate calculated in (Equation 22) is not limited to an integer but a real number.

Further, when the chromatic aberration correction is performed in the sampling coordinate setting unit 41, the deformation function f becomes a function f _R , f _G , f _B that is different for each color of _R , _G , _B, and the sampling coordinates are R, G, For each B, (Expression 23), (Expression 24), and (Expression 25). If f is the identity transformation represented by (Equation 26), the same result as in the first embodiment can be obtained.

  Next, as shown in FIG. 7, the sampling unit 53 includes four color planes (R plane, Gr plane, Gb plane, and B plane) generated by the color plane decomposition unit 21 and an R @ B generation unit. Based on the generated R @ B plane and the B @ R plane generated by the B @ R generation unit 51, the sampling value at the sampling position set by the sampling coordinate setting unit 41 (the pixel value for each color at the sampling position) Is calculated).

In other words, as the sampling coordinates, from R plane _{(u sR,} v _sR), from Gr plane _{(u sG,} v _sG), from Gb plane _{(u sG, v sG),} from the B plane _{(u sB,} v _sB), R @ from the B plane _{(u sR, v sR),} B @ from R plane _{(u sB, v sB)} is calculated.

At this time, since the sampling coordinates (u _s , v _s ) are not limited to integers (that is, (u _s , v _s ) does not necessarily match one pixel center), the sampling value is (u _s , v _s ) is calculated by linear interpolation from four valued pixels (pixel values of the same color that each color plane originally has). This interpolation is preferably performed by bilinear interpolation.

  As shown in FIG. 7H, since each color plane has four valued pixels in the form of vertical and horizontal grid points, the four valued pixels surrounding the sampling coordinates 401 to 406 have the sampling coordinates of (100. 8, 101.4), it is located at the apex of a square with a side length of 2 surrounding the sampling coordinates.

That is, for example, if the sampling coordinates (u _sR , v _sR ) for the R plane are (100.8, 101.4), the coordinates (u, v) of the four valued pixels surrounding this are (100, 100). ), (100, 102), (102, 100), (102, 102).

  Therefore, the ratio of the distances between the significant pixels facing each other via the sampling coordinates (here, the u direction is 0.4: 0.6 and the v direction is 0.7: 0.3) is obtained. Using the pixel value of the value pixel, the R pixel value at the sampling position (100.8, 101.4) is calculated by interpolation.

  The pixel values of the four valued pixels are represented by R (100, 100), R (100, 102), R (102, 100), R (102, 102), and sampling positions (100.8, 101. When the R pixel value of 4) is represented by R (100.8, 101.4), the pixel value R (100.8, 101.4) of the sampling coordinate 401 on the R plane is represented by R (100.8). , 101.4) = 0.6 * 0.3 * R (100,100) + 0.6 * 0.7 * R (100,102) + 0.4 * 0.3 * R (102,100) +0. 4 * 0.7 * R (102, 102).

Gr pixel value at the sampling coordinate 402 on the Gr plane, Gb pixel value at the sampling coordinate 403 on the Gb plane,
The B pixel value at the sampling coordinate 404 on the B plane, the R pixel value at the sampling coordinate 405 on the R @ B plane, the B pixel value at the sampling coordinate on the B @ R plane, etc. Similarly, it can be calculated by interpolation from the pixel values of four significant pixels surrounding the sampling coordinates.

  The sampling coordinate setting unit 41 sets sampling coordinates in association with all the pixels in the output color image, and the sampling unit 53 calculates sampling values at these all sampling coordinates. Then, this sampling value can be regarded as an image signal by arranging it at the corresponding pixel position of the output color image.

Next, the color generation unit 54 includes a luminance generation unit 54A that generates luminance at each pixel of the output color image and a color difference generation unit 54B that generates color differences .

  The luminance generation unit 54A calculates the pixel values of Gr, Gb, R, R @ B, B, and B @ R calculated by the sampling unit 53 for each pixel of the output color image 1: 1: (1-k). The luminance signal Y is assumed to be Y = Gr + Gb + (1-k) R + kR @ B + (1-k) B + kB @ R so as to be combined at a ratio of: k: (1-k): k. At this time, the composition ratio of the luminance signal is R: G: B = 1: 2: 1.

  As for the coefficient k used for obtaining the luminance Y, k = 2/3 is optimal for suppressing zipper artifacts, but if k = 2/3, the effect of suppressing zipper artifacts can be obtained. Since the numerical value 2/3 is expressed as 0.101010 ... b in binary number, k is in the range of 1/2 (0.1b in binary number) to 3/4 (0.11b in binary number). In this case, the binary number may be set to a simple numerical value.

Further, the sum R _tot of R and R @ B at luminance Y is expressed by an arithmetic expression of R _tot = (1-k) R + kR @ B, and is transformed into the following expression by incorporating the origin of R @ B. it can.

That is, the sum R _tot (u _sR , v _sR ) of R and R @ B is R _tot (u _sR , v _sR ) = (1−k) * R (u _sR , v _sR ) + k * R @ B ( u _sR , v _sR ) = (1−k) * R (u _sR , v _sR ) + (k / 4) * R (u _sR −1, v _sR −1) + (k / 4) * R (u _sR− 1, _vsR + 1) + (k / 4) * R ( _usR− 1, _vssR + 1) + (k / 4) * R ( _usR− 1, _vsR− 1).

For example, as shown in FIG. 7, taking the value at (u _sR , v _sR ) = (100.8, 101.4) as an example, R _tot (100.8, 101.4) = (1−k ) (0.6 * 0.3 * R (100,100) + 0.6 * 0.7 * R (100,102) + 0.4 * 0.3 * R (102,100) + 0.4 * 0. 7 * R (102,102) + (k / 4) * (0.1 * 0.8 * R (98,100) + 0.1 * 0.2 * R (98,102) + 0.9 * 0. 8 * R (100,100) + 0.9 * 0.2 * R (100,102)) + (k / 4) * (0.1 * 0.8 * R (98,102) + 0.1 * 0 .2 * R (98,104) + 0.9 * 0.8 * R (100,102) + 0.9 * 0.2 * R (100,104)) + (k / 4) * (0.1 * 0.8 * R (100, 00) + 0.1 * 0.2 * R (100,102) + 0.9 * 0.8 * R (102,100) + 0.9 * 0.2 * R (102,102)) + (k / 4 ) * (0.1 * 0.8 * R (100,102) + 0.1 * 0.2 * R (100,104) + 0.9 * 0.8 * R (102,102) + 0.9 * 0 .2 * R (102,104)) = 0.1 * 0.8 * (k / 4) * R (98,100) + 0.1 * (k / 4) * R (98,102) +0.1 * 0.2 * (k / 4) * R (98,104) + (0.8 * (k / 4) + 0.6 * 0.3 * (1-k)) * R (100,100) + ((K / 4) + 0.6 * 0.7 * (1-k)) * R (100,102) + 0.2 * (k / 4) * R (100,104) + (0.9 * 0 .8 * (k / 4) + 0.4 * 0.3 * (1-k ) * R (102,100) + (0.9 * (k / 4) + 0.4 * 0.7 * (1-k)) * R (102,102) + 0.9 * 0.2 * (k / 4) * R (102, 104).

As a result, even if R @ B is not clearly indicated in the arithmetic expression, the value of R in the 5 × 5 pixels around (u _sR , v _sR ) = (100.8, 101.4) It can be seen that the sampling value can be calculated. However, when the R sampling coordinates (u _sR , v _sR ) are exceptionally at the same horizontal and vertical positions as the B pixel, the sampling value is calculated from six or four R pixel values.

  Next, the color difference generation unit 54B generates two color difference signals C1 and C2 from the R, Gr, Gb, and B pixel values (sampling values) generated by the sampling unit 53. Here, C1 is a value obtained by subtracting the R pixel signal and the B pixel signal from the sum of the Gr pixel signal and the Gb pixel signal, while C2 is the R pixel signal and the B pixel. Represents the difference in signal.

  Further, when C1 is generated, the pixel values of R, Gr, Gb, and B are set to (−1 so that the composition ratio of R: G: B = (− 1): 2: (− 1) is obtained. ): 1: 1: (-1), and C1 = (− R + Gr + Gb−B) / 4 is calculated. On the other hand, when C2 is generated, the pixel values of R, Gr, Gb, and B are set to 2: 0: 0 :( so that the composition ratio is R: G: B = 2: 0: (− 2). -2) to calculate C2 = (2R-2B) / 4.

  Further, the color difference generation unit 54B performs correction for suppressing false colors on the color difference generated as described above. As an example of false color suppression, first, a high frequency component K in the vicinity of fs / 2 (Nyquist frequency) is calculated by an arithmetic expression of K = (2Gr−2Gb) / 4. Next, by subtracting the absolute value of K2 from the absolute value of C2 using the following formula, it is possible to suppress red and blue false colors generated in the high-frequency part. C2 → sign (C2) max (0, abs (C2) −abs (K))

  Next, the visual correction unit 34 performs predetermined processing on the luminance signal Y obtained by the luminance signal generation unit 25 and the color difference signals C1 and C2 obtained by the color difference generation unit, and a color composed of these color signals. Correct the appearance of the image. For example, low-frequency filter processing is performed on the color difference signals C1 and C2 to reduce color noise, or high-frequency enhancement is performed on the luminance signal Y to enhance resolution.

  Next, the procedure of the image processing method and the image processing program of the second embodiment will be described with reference to FIG. This procedure is executed by the CPU 18 giving a command signal to each functional unit based on a program stored in the ROM 19. Further, S in FIG. 8 represents a step.

  First, this procedure starts when an activation signal is input to the imaging apparatus 1A or the image processing apparatus 100A by the operator, and then the process proceeds to S101 and S102.

  Next, in S101, an image signal (color mosaic image) is read into the image processing apparatus 100A via the imaging unit 2A, and then the process proceeds to S201.

  In step S201, the color plane separation unit 21 is used to store the R pixel signal, the Gr pixel signal, the Gb pixel signal, and the B pixel signal in association with the Bayer array, and then the process proceeds to S301.

  Next, in S301, the R @ B generation unit 51 and the B @ R generation unit 52 are used to generate the value of the R pixel at the position of the B pixel of the color mosaic image (generate an R @ B plane), and The value of the B pixel at the position of the R pixel of the mosaic image is generated (B @ R plane is generated), and then the process proceeds to S401.

  On the other hand, in step S102, the sampling coordinate setting unit 41 is used to consider image deformation such as distortion correction, camera shake correction, and digital zoom, and output based on the parameters acquired by the aberration coefficient setting unit 39 and the shake detection unit 40. Sampling coordinates on the color mosaic image corresponding to the pixel position of the color image are set, and then the process proceeds to S401.

  Next, in S401, using the sampling unit 53, the four color planes (R plane, Gr plane, Gb plane, and B plane) generated by the color plane decomposition unit 21 and the R @ B generation unit 51 are generated. Based on the R @ B plane and the B @ R plane generated by the B @ R generation unit 52, the sampling value at the sampling position set by the sampling coordinate setting unit 41 (the pixel value for each color at the sampling position) After that, the process proceeds to S501 and S502.

  Next, in S501, using the luminance generation unit 54A, the pixel values of Gr, Gb, R, R @ B, B, and B @ R calculated by the sampling unit 53 are 1: 1: (1-k): Luminance Y is calculated by the following equation: Y = Gr + Gb + (1-k) R + kR @ B + (1-k) B + kB @ R, so that k: (1-k): k is combined. , The process proceeds to S601.

  On the other hand, in S502, the color difference generation unit 54B is used to generate two color difference signals C1 and C2 from the R, Gr, Gb, and B pixel values (sampling values) generated by the sampling unit 53, and then The process proceeds to S601.

  Next, in S601, it is determined whether or not there is a next sampling coordinate for generating the luminance Y and the color differences C1 and C2, and when there is a next target point (Yes), S401 to 601 are repeated, and the next sampling is performed. When there is no coordinate (No), the process proceeds to S701.

  In step S701, the visual correction unit 34 is used to improve the appearance of the image with respect to the color image signals (luminance Y, color difference C1, and color difference C2) generated by the luminance generation unit 54A and the color difference generation unit 54B. Image correction, conversion to YUV chromaticity coordinates, which are color space coordinates of a standard color image, and the like are performed, and then the process proceeds to S801.

  In step S801, the compression unit 35 is used to compress the digital image signal of the color image output via the visual correction unit 34 by a method such as JPEG (Joint Photographic Experts Group), and the size of the image data at the time of recording. After that, the process proceeds to S901.

  In step S901, the compressed digital image signal is stored in a recording medium such as a flash memory using the recording unit 36, and then the image processing program ends.

  As described above, according to the image processing apparatus 100A, the image processing method, and the image processing program described in the second embodiment, R, Gr, Gb, B from the color mosaic image via the color plane separation unit 21. Then, through the R @ B plane generation unit 51, the interpolation value of the R pixel surrounding the B pixel at the B pixel position is replaced with the B pixel value, and the R pixel value @B Interpolation is generated as the pixel position, and the B pixel value @R pixel is replaced by the R pixel value by replacing the interpolation value of the B pixel surrounding the R pixel at the R pixel position via the B @ R plane generation unit 52. Interpolated as a position, and then, through the sampling unit 53, at the sampling position of the mosaic image associated with the pixel position of the color image, the R plane, Gr plane, Gb plane, B plane, R pixel value Interpolate and generate pixel values for each of the B pixel position plane and B pixel value @ R pixel position plane to generate sampling values for each plane, and then synthesize the sampling values for each color plane via the luminance generation unit 54A Since the luminance of each pixel of the color image is generated, zipper artifacts appearing at the boundary between the red-based image and the blue-based image can be effectively suppressed, and furthermore, the pixel position of the color mosaic image can be suppressed. Regardless, high-quality luminance can be generated at any pixel position.

  In addition, according to the image processing apparatus 100A, the image processing method, and the image processing program described in the second embodiment, based on the blur correction based on the detection value from the blur detection unit 40 and the set value from the aberration coefficient setting unit 39. Since the sampling coordinate setting unit 41 for setting the sampling position in the input image is provided so as to correct the aberration, it is not necessary to correspond each pixel of the color mosaic image and the output color image to 1: 1, and resolution conversion is performed. And various image deformations can be applied at the same time to improve convenience. That is, according to the image processing using the luminance signal generation method of the present embodiment, it is possible to obtain a high-quality luminance signal by suppressing the occurrence of zipper artifacts and jaggies, and to perform desired image deformation. High-quality color images can be generated.

  As mentioned above, although one Example of this invention was described, this invention is not limited to the said Example, It can take a various aspect.

1 is a block diagram illustrating a configuration of an imaging apparatus in a first embodiment to which the present invention is applied. It is a figure showing the composition of a Bayer arrangement. In the first embodiment, a diagram for generating the luminance of each pixel represents the luminance Y ₀ tentative (a) is generated by the SwitchY generator, to correct the luminance Y ₀ is (b) FIG. 5C is a diagram showing a luminance correction value D generated accordingly, and FIG. 8C shows a luminance Y ₁ corrected by subtracting a predetermined amount of the luminance correction value D from the provisional luminance Y ₀ . It is a flowchart showing the procedure of the image processing method and image processing program in the said 1st Embodiment. It is a block diagram showing the structure of the imaging device in 2nd Embodiment to which this invention was applied. It is explanatory drawing which produces | generates the B @ R plane and R @ B plane in the said 2nd Embodiment. It is explanatory drawing of sampling in the said 2nd Embodiment. It is a flowchart showing the procedure of the image processing method and image processing program in the said 2nd Embodiment. In the conventional example, it is a figure explaining the generation principle of a zipper artifact, Comprising: (a) is a figure showing the blue and red image boundary in a Bayer arrangement, (b) is associated with (a) at each pixel position. The figure which gave the pixel value, (c) is the figure by which the luminance value was produced | generated for every pixel.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1,1A ... Imaging device, 2, 2A ... Imaging unit, 3 ... Imaging lens, 5 ... Imaging element, 5a ... Bayer arrangement color filter, 6 ... AFE (Analog Front End), 7 ... Correlated double sampling circuit, 8 ... variable gain amplifier (AGC), 9 ... A / D converter, 10 ... detection unit, 11 ... sensor, 12 ... lens driving unit, 13 ... TG (Timing Generator), 18 ... CPU (Central Processing Unit) ), 19... ROM (Read Only Memory), 21... Color plane separation unit, 22... R field memory, 23 a... Gr field memory, 23 b... Gb field memory, 24. SwitchY generator 27 ... RB plane generation unit, 28 ... luminance correction value generation unit, 29 ... luminance correction unit, 30 ... color difference generation unit, 31 ... C1 generation unit, 32 ... C2 generation unit, 34 ... visual correction unit, 35 ... compression unit, 36 ... Recording unit, 37 ... Range state detection unit, 38 ... Aberration coefficient storage table, 39 ... Aberration coefficient setting unit, 40 ... Blur detection unit, 41 ... Sampling coordinate setting unit, 51 ... R @ B generation unit, 52 ... B @R generator, 53 ... sampling unit, 54 ... color generator, 54A ... luminance generator, 54B ... color difference generator, 100, 100A ... image processing apparatus.

Claims (9)

A color image having a plurality of color information is generated based on a color mosaic image output through an image sensor having a color filter with a Bayer array of three colors of R (red), G (green), and B (blue). An image processing apparatus that
A predetermined pixel range including a target point for luminance generation is set on the mosaic image, and the luminance for generating the luminance at the target pixel based on the R, G, and B pixel values located in the pixel range is set. A generating means,
The luminance generating means;
A first luminance generation unit that applies a low-pass filter to the color mosaic image to remove a color carrier and generates a first luminance value for each pixel;
BR plane generating means for generating a BR plane composed of B and R pixels excluding the G pixel from the color mosaic image;
Correction value generating means for filtering the BR plane by applying a band filter, and obtaining an output value for each pixel output through the band filter as a luminance correction value;
A second luminance generating unit that generates a second luminance value by adding the correction value generated by the correction value generating unit to the first luminance value generated by the first luminance generating unit;
The second luminance value is a luminance value in the color image;
An image processing apparatus. The low-pass filter is a convolution with a 3 × 3 pixel kernel,
The image processing apparatus according to claim 1. The bandpass filter is a convolution with a 5 × 5 pixel kernel,
The image processing apparatus according to claim 1 or 2, wherein A color image having a plurality of color information is generated based on a color mosaic image output through an image sensor having a color filter with a Bayer array of three colors of R (red), G (green), and B (blue). An image processing method for
A predetermined pixel range including a target point for luminance generation is set on the mosaic image, and the luminance for generating the luminance at the target pixel based on the R, G, and B pixel values located in the pixel range is set. Using the generation step,
In the luminance generation step,
A first luminance generation step of applying a low-pass filter to the color mosaic image to remove a color carrier and generating a first luminance value for each pixel;
A BR plane generation step of generating a BR plane composed of B and R pixels excluding the G pixel from the color mosaic image;
A correction value generation step of filtering the BR plane by applying a band filter, and obtaining an output value for each pixel output through the band filter as a luminance correction value;
A second luminance generation step of generating a second luminance value by adding the correction value generated in the correction value generation step to the first luminance value generated in the first luminance generation step;
Use
The second luminance value is a luminance value in the color image.
An image processing method. The low-pass filter is a convolution with a 3 × 3 pixel kernel,
The image processing method according to claim 4. The bandpass filter is a convolution with a 5 × 5 pixel kernel,
6. The image processing method according to claim 4 or 5, wherein: A color image having a plurality of color information is generated based on a color mosaic image output through an image sensor having a color filter with a Bayer array of three colors of R (red), G (green), and B (blue). An image processing program for
A predetermined pixel range including a target point for luminance generation is set on the mosaic image, and the luminance for generating the luminance at the target pixel based on the R, G, and B pixel values located in the pixel range is set. Using the generation step,
In the luminance generation step,
A first luminance generation step of applying a low-pass filter to the color mosaic image to remove a color carrier and generating a first luminance value for each pixel;
A BR plane generation step of generating a BR plane composed of B and R pixels excluding the G pixel from the color mosaic image;
A correction value generation step of filtering the BR plane by applying a band filter, and obtaining an output value for each pixel output through the band filter as a luminance correction value;
The correction value generated in the correction value generation step is added to the first luminance value generated in the first luminance generation step to generate a second luminance value, and the second luminance value is used as the color image. A second luminance generation step for generating as a luminance value in
An image processing program for causing a computer to execute. The low-pass filter is a convolution with a 3 × 3 pixel kernel,
The image processing program according to claim 7. The bandpass filter is a convolution with a 5 × 5 pixel kernel,
The image processing program according to claim 7 or 8 , wherein