JP5454820B2 - Image processing apparatus, image processing method, and image processing program - Google Patents

Description

  The present invention relates to an image processing apparatus and image processing for generating a color image from a color mosaic image output via an image pickup device having a color filter with a Bayer array of three colors of R (red), G (green), and B (blue). More particularly, the present invention relates to a technique for generating a luminance signal from an output signal output via an image sensor.

  2. Description of the Related Art Conventionally, in an imaging system such as a digital camera, an image processing apparatus and an image processing method are known in which a subject image is formed on an image sensor via a lens, the subject image is photoelectrically converted by the image sensor, and an image signal is generated. It has been.

  As a single-plate image sensor, a plurality of photoelectric conversion elements are provided in a matrix, and a color filter is provided in front of the image sensor, and image processing is performed by applying signal processing to the pixel signals of each color output via the color filter. An image processing apparatus and an image processing method to be generated are known.

  In a so-called single-plate image sensor using only one image sensor, specific color filters corresponding to each pixel on the image sensor, such as red (R), green (G), and blue (B) colors, are used. A filter is provided. Usually, a pixel provided with an R color filter is an R pixel (red pixel), a pixel provided with a G color filter is G pixel (green pixel), and a pixel provided with a B color filter is B pixel (blue pixel). Pixel).

  A color filter called a primary color Bayer array is used as a representative color filter in a single-plate image sensor. The color filter of the primary color Bayer arrangement is such that rows in which R and G are alternately arranged in the horizontal direction and rows in which G and B are alternately arranged in the horizontal direction are alternately arranged in the vertical direction. A plurality of basic blocks each having two vertical pixels (2 × 2 pixels) as one unit (one unit) are periodically arranged. Each basic block has a structure in which two green pixels are arranged on one diagonal and a red pixel and a blue pixel are arranged on the other diagonal.

  In some image processing apparatuses, a luminance signal and a color difference signal are generated from an image signal output via a single-plate image sensor, and an image signal of a color image is generated from these signals (for example, Patent Document 1).

  Specifically, in a single-plate image sensor, each pixel has only single color information, but in order to display a color image, all the values of red (R), green (G), and blue (B) are each pixel. Is necessary. For this reason, in image processing using a single-plate image sensor, so-called demosaic processing is performed based on a color mosaic image in which each pixel has only one of R, G, and B components. Here, the demosaic process is an interpolation operation using the luminance information of other insufficient colors collected from the surrounding pixels for the single color information of each pixel of the color mosaic image, so that each pixel is R, This is processing for generating a color image having all of the G and B components (so-called color interpolation processing).

In addition, as a simple method for generating a luminance signal, a color carrier component is obtained by directly filtering a color mosaic image output from an image sensor using, for example, a low-pass filter represented by (Equation 1). There is known a method of removing luminance and generating luminance using a pixel signal obtained through a filter (see, for example, Patent Document 2). That is, a pixel signal output through a uniform low-pass filter for a color mosaic image is used as a luminance signal (hereinafter referred to as SwitchY luminance generation method).

JP 2000-341702 A JP 2000-287219 A

  However, according to the conventional method of generating luminance by applying a low-pass filter to a predetermined pixel range centering on the pixel of interest, light and darkness is jagged, particularly along the image boundary between an oblique red image and a blue image. Pattern zipper artifacts occurred and there was room for further improvement. Here, the zipper artifact refers to a phenomenon in which light and darkness that does not exist in the original image appear in a zipper shape (a so-called slide fastener shape) along the image boundary by demosaic processing.

  Specifically, as shown in FIG. 9A, blue having a luminance of (R, G, B) = (0, 0, 1) in the upper right region (RU in the figure) via the image boundary L. When an image is formed and a red image having a luminance of (R, G, B) = (1, 0, 0) is formed in the lower left region (LD in the figure), the color mosaic image is shown in FIG. As shown in (b), B pixel is 1 in the image range in the upper right region, R pixel is 1 in the image range in the lower left region, and all other pixels are 0.

  Then, when the image signal shown in FIG. 9B is subjected to the low-pass filter of (Equation 1) to remove the color carrier component, as shown in FIG. The pixel values 3/16 and 5/16, which are undesirable, are alternately generated, and a light and dark pattern called a zipper artifact is generated. Further, such a luminance mismatch at the image boundary is not limited to the boundary between red and blue, but easily occurs in an area where the color difference between red and blue changes, and appears as jagged in the color image.

  Therefore, the present invention suppresses zipper artifacts and jaggies that occur along the image boundary between red and blue when generating a color image from a mosaic image output from a single-plate imaging device with a primary color Bayer array, An object of the present invention is to provide an image processing apparatus and an image processing method capable of obtaining a quality color image.

  The invention according to claim 1, which has been made to achieve such an object, is output via an image pickup device having a color filter with a Bayer array of three colors of R (red), G (green), and B (blue). An image processing apparatus for generating a color image having a plurality of color information for each pixel from a color mosaic image in which each pixel has color information of a single color, and in the Bayer array, adjacent to R in the horizontal direction A color plane separation unit that separates the color mosaic image into R, Gr, Gb, and B pixels when the G arranged in G is represented as Gr and the G arranged next to B is represented as Gb. An R @ B plane generation unit that interpolates and generates an R pixel value @ B pixel position by replacing the interpolation value of the R pixel surrounding the B pixel with the B pixel value at the B pixel position; Of the B pixel surrounding the R pixel at the pixel position of A B @ R plane generating unit that interpolates and generates a B pixel value @R pixel position by replacing the value with a R pixel value, and the R plane at the sampling position of the mosaic image associated with the pixel position of the color image, A pixel value is interpolated and generated for each of the Gr plane, Gb plane, B plane, R @ B plane, and B @ R plane, and a sampling means for generating a sampling value for each plane is generated via the sampling means. Luminance generating means for generating a luminance of each pixel of the color image by combining sampling values of the R plane, Gr plane, Gb plane, B plane, R @ B plane, and B @ R plane It is characterized by having.

  According to the image processing device of claim 1, in the color mosaic image, the interpolation value of the R pixel surrounding the B pixel is replaced with the B pixel value at the B pixel position, and the R pixel value @ B pixel An R @ B plane generation unit that generates an interpolation as a position, and an B that generates an interpolation as a B pixel value @ R pixel position by replacing an interpolation value of a B pixel that surrounds the R pixel with an R pixel value at an R pixel position At the sampling position of the mosaic image associated with the @R plane generation unit and the pixel position of the color image, each of the R plane, the Gr plane, the Gb plane, the B plane, the R @ B plane, and the B @ R plane Sampling means for interpolating pixel values to generate sampling values for each plane, and the R plane, Gr plane, Gb plane, B plane, R @ B generated via the sampling means And a luminance generation means for generating the luminance of each pixel of the color image by synthesizing the sampling values of the lane and the B @ R plane, so that zipper artifacts appearing at the boundary between the red image and the blue image It can be effectively suppressed, and furthermore, high-quality luminance can be generated at any pixel position regardless of the pixel position of the color mosaic image.

  Further, according to the first aspect of the present invention, as in the second aspect of the present invention, the luminance generation means uses a ratio of approximately 1: 2 between the sampling value of the R plane and the sampling value of the R @ B plane. By combining with, zipper artifacts appearing at the red image boundary can be effectively suppressed.

  Further, according to the first aspect of the present invention, as in the third aspect of the present invention, the luminance generation means uses a ratio of approximately 1: 2 between the sampling value of the B plane and the sampling value of the B @ R plane. By combining with, zipper artifacts appearing at the blue image boundary can be effectively suppressed.

  Next, the invention described in claim 4 is outputted through an image sensor having a Bayer array color filter of R (red), G (green), and B (blue), and each pixel has a single color. An image processing method for generating a color image having a plurality of pieces of color information for each pixel from a color mosaic image having color information, wherein the G is arranged next to the R in a horizontal direction in the Bayer array. A color plane separation step for separating the color mosaic image into R, Gr, Gb, and B pixels, and G pixels arranged as Gr and G as Gb. R @ B plane generating step of interpolating and generating an R pixel value @B pixel position by replacing the interpolation value of the R pixel surrounding the B pixel with the B pixel value at the position, and at the R pixel position, The interpolated value of the B pixel surrounding the R pixel is the R pixel value. At the B @ R plane generation step of interpolating and generating the B pixel value @R pixel position by replacement, and at the sampling position of the mosaic image associated with the pixel position of the color image, the R plane, Gr plane, Gb plane r− A sampling step for interpolating and generating pixel values for each of the B plane, R @ B plane, and B @ R plane, and generating sampling values for each plane, and the R plane generated through the sampling step, A luminance generation step of generating the luminance of each pixel of the color image by combining the sampling values of the Gr plane, the Gb plane, the B plane, the R @ B plane, and the B @ R plane. .

  According to the image processing method of claim 4, a color plane separation step for separating a color mosaic image into R, Gr, Gb, and B pixels, and an R that surrounds the B pixels at the B pixel positions. An R @ B plane generation step of interpolating and generating an R pixel value @B pixel position by replacing the interpolation value of the B pixel with the B pixel value, and an interpolation value of the B pixel surrounding the R pixel at the R pixel position In the B @ R plane generation step of interpolating and generating the pixel value of R as the B pixel value @R pixel position, and the sampling position of the mosaic image associated with the pixel position of the color image, the R plane, the Gr plane, A sampling step for interpolating and generating pixel values for each of the Gb plane, B plane, R @ B plane, and B @ R plane, and generating a sampling value for each plane, and a sampling step Generating step of generating luminance of each pixel of a color image by synthesizing sampling values of R plane, Gr plane, Gb plane, B plane, R @ B plane, and B @ R plane generated Thus, similarly to the invention according to claim 6, zipper artifacts appearing at the boundary between the red image and the blue image can be effectively suppressed, and further, the color mosaic image Regardless of the pixel position, high-quality luminance can be generated at any pixel position.

  Next, in the image processing method according to claim 4, as in the invention according to claim 5, in the luminance generation step, the sampling value of the R plane and the sampling value of the R @ B plane are approximately 1: 2. By synthesizing at the ratio, the zipper artifact appearing at the red-based image boundary can be effectively suppressed, as in the second aspect of the invention.

  According to a fourth aspect of the present invention, as in the sixth aspect of the present invention, the luminance generation step uses a ratio of approximately 1: 2 between the sampling value of the B plane and the sampling value of the B @ R plane. In the same manner as in the third aspect of the invention, the zipper artifact appearing at the blue image boundary can be effectively suppressed.

  Next, according to the seventh aspect of the present invention, each pixel is output through an image sensor having a Bayer array color filter of three colors of R (red), G (green), and B (blue). An image processing program for generating a color image having a plurality of color information for each pixel from a color mosaic image having color information, wherein the G is arranged next to the R in a horizontal direction in the Bayer array. A color plane separation step for separating the color mosaic image into R, Gr, Gb, and B pixels, and G pixels arranged as Gr and G as Gb. R @ B plane generating step of interpolating and generating an R pixel value @B pixel position by replacing the interpolation value of the R pixel surrounding the B pixel with the B pixel value at the position, and at the R pixel position, The interpolated value of the B pixel surrounding the R pixel is R In the B @ R plane generation step of interpolating and generating the B pixel value @R pixel position instead of the pixel value, and at the sampling position of the mosaic image associated with the pixel position of the color image, the R plane, Gr plane, Gb A sampling step for interpolating and generating pixel values for each of the plane, the B plane, the R @ B plane, and the B @ R plane, and generating a sampling value for each plane, and the sampling step generated through the sampling step A luminance generation step of generating the luminance of each pixel of the color image by synthesizing the sampling values of the R plane, Gr plane, Gb plane, B plane, R @ B plane, and B @ R plane. It is characterized by making it.

  According to the image processing program of claim 7, a color plane separation step for separating the color mosaic image into R, Gr, Gb, and B pixels, and R's surrounding the B pixels at the B pixel positions. R @ B plane generation step of interpolating and generating an R pixel value @B pixel position by replacing the pixel interpolation value with the B pixel value, and an interpolation value of the B pixel surrounding the R pixel at the R pixel position At the B @ R plane generation step of interpolating and generating as the B pixel value @ R pixel position by replacing with the R pixel value, and at the sampling position of the mosaic image associated with the pixel position of the color image, the R plane, Gr plane, Gb Sampling step for interpolating and generating pixel values for each plane, B plane, R @ B plane, and B @ R plane, and generating sampling values for each plane, and sampling Luminance generation step of generating luminance of each pixel of a color image by synthesizing sampling values of R plane, Gr plane, Gb plane, B plane, R @ B plane, and B @ R plane generated through the step By causing the computer to execute, similarly to the invention according to claim 6, zipper artifacts appearing at the boundary between the red image and the blue image can be effectively suppressed. Regardless of the pixel position of the image, high-quality luminance can be generated at any pixel position.

  According to a seventh aspect of the present invention, in the luminance generation step, the R plane sampling value and the R @ B plane sampling value are in a ratio of approximately 1: 2, as in the eighth aspect of the invention. In the same manner as in the second aspect of the invention, the zipper artifact appearing at the red image boundary can be effectively suppressed.

  Further, according to a seventh aspect of the invention, as in the ninth aspect of the invention, the luminance generation step uses a ratio of approximately 1: 2 between the sampling value of the B plane and the sampling value of the B @ R plane. In the same manner as in the third aspect of the invention, the zipper artifact appearing at the blue image boundary can be effectively suppressed.

  The image processing apparatus, the image processing method, and the image processing program according to the present invention replace the interpolation value of the R pixel that surrounds the B pixel with the B pixel value at the B pixel position in the color mosaic image. An R @ B plane generation unit that generates an interpolation as the @B pixel position, and an interpolation value of the B pixel that surrounds the R pixel is replaced with an R pixel value at the R pixel position, and is interpolated as a B pixel value @R pixel position. At the sampling position of the mosaic image associated with the pixel position of the color image and the B @ R plane generation unit to generate, the R plane, the Gr plane, the Gb plane, the B plane, the R @ B plane, and the B @ R plane The pixel value is interpolated and generated for each of the sampling means for generating the sampling value for each plane, and the R plane, Gr plane, and Gb pre-generated via the sampling means. , B plane, R @ B plane, B @ R plane sampling values are combined to generate luminance of each pixel of the color image, so that the red image and the blue image Zipper artifacts appearing at the border of the image can be effectively suppressed, and high-quality luminance can be generated at any pixel position regardless of the pixel position of the color mosaic image.

  The image processing apparatus, the image processing method, and the image processing program of the present invention synthesize the sampling value of the R plane and the sampling value of the R @ B plane at a ratio of approximately 1: 2 when generating the luminance. Thus, zipper artifacts appearing at the red image boundary can be effectively suppressed, and the blue image is synthesized by synthesizing the sampling value of the B plane and the sampling value of the B @ R plane at a ratio of approximately 1: 2. Zipper artifacts appearing at the boundary can be effectively suppressed.

1 is a block diagram illustrating a configuration of an imaging apparatus in a first embodiment to which the present invention is applied. It is a figure showing the composition of a Bayer arrangement. In the first embodiment, a diagram for generating the luminance of each pixel represents the luminance Y ₀ tentative (a) is generated by the SwitchY generator, to correct the luminance Y ₀ is (b) FIG. 5C is a diagram showing a luminance correction value D generated accordingly, and FIG. 8C shows a luminance Y ₁ corrected by subtracting a predetermined amount of the luminance correction value D from the provisional luminance Y ₀ . It is a flowchart showing the procedure of the image processing method and image processing program in the said 1st Embodiment. It is a block diagram showing the structure of the imaging device in 2nd Embodiment to which this invention was applied. It is explanatory drawing which produces | generates the B @ R plane and R @ B plane in the said 2nd Embodiment. It is explanatory drawing of sampling in the said 2nd Embodiment. It is a flowchart showing the procedure of the image processing method and image processing program in the said 2nd Embodiment. In the conventional example, it is a figure explaining the generation principle of a zipper artifact, Comprising: (a) is a figure showing the blue and red image boundary in a Bayer arrangement, (b) is associated with (a) at each pixel position. The figure which gave the pixel value, (c) is the figure by which the luminance value was produced | generated for every pixel.

(First embodiment)
Next, an embodiment of an image processing apparatus, an image processing method, and an image processing program according to the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram showing the configuration of an imaging apparatus according to the first embodiment to which the present invention is applied, FIG. 2 is a diagram showing the configuration of a color mosaic image with a Bayer arrangement in the same embodiment, and FIG. a diagram of a luminance generation in the form, (a) represents a luminance Y ₀ tentative generated by SwitchY generator represents the correction value D (b) is generated so as to correct the luminance Y _0, (c) is a diagram showing a luminance Y ₁ that is corrected by subtracting a predetermined amount of the correction value D from the luminance Y ₀ tentative. FIG. 4 is a flowchart showing the procedure of the image processing method and the image processing program in the embodiment.

  As illustrated in FIG. 1, the imaging apparatus 1 includes an imaging unit 2 that guides a subject image to the imaging element 5 and outputs the image as a digital image signal C (a mosaic image signal), and outputs the image via the imaging unit 2. The image processing apparatus 100 generates a color image having a plurality of pieces of color information for each pixel based on the digital image signal C.

  The imaging unit 2 is output from the imaging lens 3 that guides the subject image P to the imaging device 5, the imaging device 5 that converts the received imaging light into an electrical quantity and outputs the same (for example, CCD: Charge Coupled Devices) 5, and the imaging device 5. AFE (Analog Front End) 6 that converts the analog image signal to be converted into a digital image signal C and outputs it, TG (Timing Generator) 13 that controls the image sensor 5 and AFE 6 at a predetermined cycle, and the optical axis direction of the imaging lens 3 ( A lens driving unit 12 that performs slide driving in the Z direction in FIG. 3, a detection unit 10 that detects the sliding amount of the imaging lens 3 in the Z direction via the sensor 11, and the like are provided. Note that the present invention is not limited to the presence or absence of the imaging lens 3, the lens driving unit 12, the sensor 11, the detection unit 10, and the like, and as will be described later, the imaging device 5 having a color filter with an RGB three-color Bayer array. This relates to the image processing of the color mosaic image output via.

  The imaging element 5 is configured by arranging a plurality of photoelectric conversion elements in a matrix, and is configured to photoelectrically convert an imaging signal for each photoelectric conversion element and output an analog image signal.

  The image sensor 5 includes a color filter 5a having a Bayer array of three colors R (red), G (green), and B (blue) in association with the photoelectric conversion elements, and the amount of light that has passed through the filter portions of the respective colors. Is converted into an electrical signal.

  The AFE 6 is a correlated double sampling circuit (CDS: Correlated Double Sampling) 7 that removes noise from the analog image signal output via the image sensor 5, and an image that has been correlated double sampled by the correlated double sampling circuit 7. A variable gain amplifier (AGC) 8 that amplifies the signal, an A / D converter 9 that converts an analog image signal from the image sensor 5 input through the variable gain amplifier 9 into a digital image signal, and the like The image signal configured and output from the image sensor 5 is converted into a digital image signal C in association with the photoelectric conversion element and output to the image processing apparatus 100.

  In the imaging unit 2, a CMOS (Complementary Metal Oxide Semiconductor) sensor may be used instead of the imaging device 5, the correlated double sampling circuit 7, the variable gain amplifier 8, the A / D converter 9, and the like. Since the signal for each pixel output from the image pickup device 5 has only single color information, a mosaic image signal is output from the image pickup unit 2 to the image processing apparatus 100.

  Next, the image processing apparatus 100 separates the mosaic image output from the imaging unit 2 for each of R, Gr, Gb, and B pixels, and the pixel signal output from the color plane decomposition unit 21 A luminance generation unit 25 for generating luminance as a color image signal at a pixel position of the color image based on the color difference and a color difference generation unit 30 for generating color difference are provided.

  The image processing apparatus 100 also performs a known gamma correction, saturation correction, and edge enhancement for improving the appearance of the color image with respect to the color image signals output from the luminance generation unit 25 and the color difference generation unit 30. The color image output through the correction unit 34 and the visual correction unit 34 is compressed by a method such as JPEG, and the color image output through the compression unit 35 is stored in a recording medium such as a flash memory. Recording unit 36, CPU (Central Processing Unit) 18, ROM (Red Only Memory) 19 and the like. The CPU 18 performs each process of the image processing apparatus 100 and the imaging apparatus 1 according to a control program stored in the ROM 19. Control.

  The color plane separation unit 21 is associated with the Bayer array illustrated in FIG. 2 and stores an R field memory 22 that stores R pixel signals and a pixel signal of Gr (R adjacent to R in one direction). A Gr field memory 23a, a Gb field memory 23b that stores a pixel signal of Gb (R adjacent to B in one direction), and a B field memory 24 that stores a pixel signal of B. Based on the above, these pixel signals are output to the luminance generation unit 25 and the color difference generation unit 30. Note that each color field memory is a virtual object, and in actuality, each color field may be assigned by address so that each color field can be distinguished on a single memory.

  Next, the luminance generation unit 25 sets a pixel range including the target point (target pixel) for each of R, G, and B in the arrangement of the color mosaic images, and sets the R, G, and B positions in this pixel range. The luminance of the target point is generated based on the pixel value.

Luminance generation section 25, SwitchY generator 26 for generating a luminance Y ₀ provisional sum multiplied by a predetermined coefficient to the pixel value of the pixels surrounding the target pixel and the target pixel, the color mosaic image except for pixels of G An RB plane generation unit 27 that generates an RB plane consisting only of R and B pixel values, a luminance correction value generation unit 28 that generates a luminance correction value D by applying a predetermined bandpass filter to the RB plane, and provisional luminance A luminance correction unit 29 that generates a second luminance Y ₁ by subtracting an amount obtained by multiplying the correction value D by a predetermined coefficient from Y ₀ is configured.

Specifically, in the SwitchY generation unit 26, each pixel value located in the 3 × 3 pixel range centered on the pixel of interest in the color mosaic image is multiplied by the low-pass filter expressed by (Equation 1), and the pixel values are totaled. luminance Y ₀ provisional of the pixel of interest is calculated. That is, three pixels in the uv direction centered on the target pixel are set and weighted according to each pixel position, and the sum of these is set as the temporary luminance Y ₀ of the target pixel. In this embodiment, as shown in FIG. 9A, the color mosaic image has a luminance of (R, G, B) = (0, 0, 1) in the upper right region via the image boundary L. And a blue and red image are formed so as to have a luminance of (R, G, B) = (1, 0, 0) in the lower left region, and will be described below. .

In SwitchY generator 26, the luminance _{Y 0} provisional as shown in FIGS. 3 (a) is determined. In FIG. 3A, 1/16 of the pixel value entered at each pixel position is a provisional luminance Y ₀ (so-called first luminance in the present invention). For example, the luminance Y ₀ of the pixel A ₁₁ located at (u, v) = (1, 1) in FIG. 3A is Y ₀ = (A ₀₀ × 1 + A ₁₀ × 2 + A ₂₀ × 1 + A ₀₁ × 2 + A ₁₁ × _{4 + a 21 × 2 + a} 02 × 1 + a 12 × 2 + a 22 × 1) / 16 is calculated by the calculation equation, _{a 00} = ₀ on the calculation equation with reference to FIG. 9 (b), a 10 = 0, a 20 = 0 Substituting A ₀₁ = 0, A ₁₁ = 0, A ₂₁ = 1, A ₀₂ = 0, A ₁₂ = 1, and A ₂₂ = 0, Y ₀ = 4/16 is obtained.

  In the RB plane generation unit 27, since the pixel value located at the G pixel in the color mosaic image is 0, the B pixel in the upper right region and the lower left region are passed through the image boundary L as in FIG. 9B. An RB plane having a pixel value at the position of the R pixel in the region is generated.

Next, in the luminance correction value generation unit 28, each pixel value located within the 5 × 5 pixel range with the target pixel as the center in the RB plane generated by the RB plane generation unit 27 is expressed by (Expression 2). The pixel values are summed by applying a bandpass filter, and the luminance correction value D at the target pixel is obtained. That is, five pixels in the uv direction centered on the target pixel are set and weighted according to the pixel position, and the sum of these is set as the correction value D of the target pixel.

Then, as shown in FIG. 3B, the correction value D is generated for the B pixel and the R pixel. In FIG. 3B, 1/64 of the numerical value entered at each pixel position is the correction value D. For example, the correction value D of the pixel A ₃₂ located at (u, v) = (3, 2) in FIG. 3A is D = (A ₁₀ × (−1) + A ₂₀ × (−2) + A ₃₀ × (−2) + A ₄₀ × (−2) + A ₅₀ × (−1) + A ₁₁ × (−2) + A ₂₁ × (0) + A ₃₁ × (4) + A ₄₁ × (0) + A ₅₁ × (−2 ) + A ₁₂ × (−2) + A ₂₂ × (4) + A ₃₂ × (12) + A ₄₂ × (4) + A ₅₂ × (−2) + A ₁₃ × (−2) + A ₂₃ × (0) + A ₃₃ × ( 4) + A ₄₃ × (0) + A ₅₃ × (−2) + A ₁₄ × (−1) + A ₂₄ × (2) + A ₃₄ × (2) + A ₄₄ × (−2) + A ₅₄ × (−1)) / 64, and the pixel values at the pixel positions A ₂₁ , A ₄₁ , A ₁₂ , A ₄₃ , A ₁₄ , and A ₃₄ are set to 1 with reference to FIG. 9B. If the pixel values at other pixel positions are substituted as 0, a numerical value (−5) / 64 is obtained.

At this time, the bandpass filter expressed by (Expression 2) may be applied by being divided into two stages of filters as expressed by (Expression 3).

Next, the brightness correcting unit 29, by subtracting the amount obtained by multiplying the coefficient k = 2/3 in the correction value from the provisional luminance _{Y 0} generated by SwitchY generator 26, as shown in FIG. 3 (c) second obtaining luminance _{Y 1.}

In FIG. 3 (c), 1/96 of numbers entered in each pixel position is the second luminance Y _1. Here, the provisional luminance Y ₀ (FIG. 3A) generated by the SwitchY generation unit 26 and the second luminance Y ₁ corrected by the luminance correction unit 29 (FIG. 3C) are used. By comparison, it can be seen that along the image boundary L, the luminance change of the luminance Y ₁ is less than the temporary luminance Y ₀ and the zipper artifact is suppressed.

  Regarding the coefficient k, k = 2/3 is optimal for suppressing zipper artifacts, but if the range is 0 <k <(4/3), the effect of suppressing zipper artifacts can be obtained. Since the numerical value 2/3 is expressed as 0.101010 ... b in binary number, k is in the range of 1/2 (0.1b in binary number) to 3/4 (0.11b in binary number). In this case, the binary number may be set to a simple numerical value.

Further, the luminance generation unit 25 obtains the correction value D for the RB plane using (Equation 2), and subtracts the amount obtained by multiplying the correction value D by a coefficient from the temporary luminance Y ₀ to obtain the second luminance Y. was calculated _1, these alternative, for the pixel position of the G color mosaic calculates a second luminance Y ₁ using (equation 4), to the pixel position of the B and R ( equation 5) second may calculate the luminance Y ₁ using. As a result, the second luminance Y ₁ calculated by the second luminance Y ₁ and calculated by the luminance correction section 29 (Equation 4) and (5) have the same value.

In (Expression 4) and (Expression 5), the 5 × 5 matrix on the left side is obtained by setting the value located at the G pixel to 0, and if this 0 pixel is ignored, the pixel of interest is the center. Four or five G pixels present in the peripheral 3 × 3 pixels and four or nine R and B pixels present in the peripheral 5 × 5 pixels are used for calculating the second luminance Y ₁ . .

Also, when calculating the second brightness Y _1, the G plane obtained by extracting only G pixels multiplied by the filter (Equation 1), whereas, for the RB plane obtained by extracting only the R and B pixel (formula multiplied by the filter 6), may be the sum of the pixel values output through both filters as the second luminance Y _1.

Further, the filter of (Expression 6) may be configured by a two-stage filter as expressed in (Expression 7).

  Next, the color difference generation unit 30 generates a color difference C1 and a color difference C2 that are color components of the target pixel using each pixel value included in a predetermined pixel range centered on the target pixel. Here, C1 represents a value obtained by subtracting the R pixel signal and the B pixel signal from the sum of the Gr pixel signal and the Gb pixel signal, while C2 represents the R pixel signal and the B pixel signal. Represents the difference between Specifically, in a predetermined pixel range, C1 is generated so that the composition ratio of R: G: B = (− 1): 2: (− 1), while R: G: B = 2: C2 is generated so that the composition ratio is 0: (− 2).

An example of generation of C1 and C2 is described below. First, in the pixel range centered on the target pixel, the filter of (Equation 8) is applied to the color mosaic image when the target pixel is the G pixel, and to the color mosaic image when the target image is the B and R pixels ( The filter of Expression 9) is applied, and the result is set as the color difference C1 at the target pixel position.

Further, in the pixel range centered on the target pixel, the filter of (Equation 10) is applied to the color mosaic image for the R pixel, and the filter of (Equation 11) is applied to the color mosaic image for the Gr pixel. For the Gb pixel, the filter of (Equation 12) is applied to the color mosaic image, and for the B pixel, the filter of (Equation 13) is applied to the color mosaic image, and the result is the color difference C2 at the target pixel position. Note that the present invention is an invention relating to luminance generation, and is not limited to generation of color differences.

Next, vision correcting unit 34 adds a predetermined processing on the luminance signal Y ₁ and the color difference signals C1, C2 obtained by the color difference generation unit obtained by the luminance signal generating unit 25, consisting of the color signals Correct the appearance of the color image. For example, reducing the false color or color noise by applying a low-frequency filtering on the color difference signals C1, C2, or emphasize a sense of resolution by performing high frequency enhancement on the luminance signal Y _1.

Note that the luminance signal Y ₁ generated by the embodiment of the present invention has a modulation factor near fs / 4 (fs is a sampling frequency) in a black and white subject, as compared with the luminance signal Y ₀ generated by the SwitchY method. Therefore, it is desirable to perform high frequency emphasis so as to increase the degree of modulation near fs / 4 via the visual correction unit 34.

Further, when the compression unit 35 compresses the image signal, the visual correction unit 34 uses the YUV chromaticity coordinates as the color space coordinates of the standard color image, so the luminance signal Y ₀ generated by the luminance generation unit 25 is used. The color differences C1 and C2 generated by the color difference generation unit 35 are converted into YUV values (Y is the luminance of the pixel, U is the color difference between the luminance Y and blue (B), and V is the luminance Y and red (R)). Color difference). For more conversion, because of the relationship between the luminance signal Y ₀ and color difference C1 generated by the color difference generation unit 30, C2 which is generated by the luminance generation section 25 (Equation 14), the YUV values using equation (15) I do.

  Next, the procedure of the image processing method and the image processing program of the first embodiment will be described with reference to FIG. This procedure is executed by the CPU 18 giving a command signal to each functional unit based on a program stored in the ROM 19. Further, S in FIG. 4 represents a step.

  First, this procedure starts when an activation signal is input to the imaging apparatus 1 or the image processing apparatus 100 by an operator.

  Next, in S100, an image signal (color mosaic image) is read into the image processing apparatus 100 via the imaging unit 2, and the R plane signal, the Gr pixel signal, and the Gb pixel are associated with the Bayer array by the color plane decomposition unit 21. Each signal and B pixel signal is stored, and then the process proceeds to S200.

  Next, in S200, a target point (target pixel) is set corresponding to the pixel arrangement of the color mosaic image, and then the process proceeds to S300. At this time, the target point to be set is any pixel of the color mosaic image, and is determined by a program stored in the ROM 19.

Next, in S300, the luminance of the target point (target pixel) is generated. Specifically, as illustrated in FIG. 4B, in S310, the SwitchY generation unit 26 is used, and each pixel value positioned in a 3 × 3 matrix centered on the pixel of interest in the color mosaic image is expressed by (Expression 1). by applying a low-pass filter that represents the sum of the pixel values to obtain the luminance Y ₀ of the temporary target pixel, then it goes to S340.

  On the other hand, in S320, the RB plane generating unit 27 is used to generate an RB plane in which the pixel value located at the G pixel in the color mosaic image is 0 and only the R and B pixels have pixel values, and then the process proceeds to S330.

  Next, in S330, the brightness correction value generation unit 28 is used, and in the RB plane generated by the RB plane generation unit 27, each pixel value located in a 5 × 5 matrix with the target pixel as the center is expressed by (Expression 2). The correction value D at the target pixel is generated by applying the bandpass filter, and then the process proceeds to S340.

Then, in S340, using the luminance correction unit 29 obtains a luminance _{Y 1} by subtracting the amount obtained by multiplying a predetermined coefficient k from the luminance _{Y 0} tentative generated in the correction value D generated at S330 in S310, then, FIG. The process proceeds to S500 of 4 (a).

  Next, in S500, it is determined whether or not there is a next target point (target pixel) for generating luminance. When there is a next target point (Yes), S200 to 500 are repeated, and the next target point ( When there is no (pixel of interest) (No), the process proceeds to S600.

Then, in S600, using a vision correcting unit 34, the color image signal generated by the luminance generation section 25 and a color difference generating section 30 (luminance Y _1, chrominance C1, chrominance C2) with respect to, for improving the appearance of the image Image correction, conversion to YUV chromaticity coordinates, which are color space coordinates of a standard color image, and the like, and then the process proceeds to S700.

  Next, in S700, the digital image signal of the color image output via the visual correction unit 34 is compressed by a method such as JPEG (Joint Photographic Experts Group) using the compression unit 35, and the size of the image data at the time of recording is recorded. After that, the process proceeds to S800.

  In step S800, the compressed digital image signal is stored in a recording medium such as a flash memory using the recording unit 36, and then the image processing program ends.

As described above, according to the image processing apparatus 100 and the image processing method and image processing program according to the first embodiment, the color mosaic image, and generates a luminance Y ₀ provisional using SwitchY generator 26 , using the RB plane generating unit 27 and the luminance correction value generation unit 28 generates the correction value D, then, using the luminance correction unit 29 subtracts the correction value from the luminance Y _0, the luminance Y which suppresses zipper artifact ₁ can be generated, and a high-quality color image can be obtained.

(Second Embodiment)
Next, a second embodiment of the present invention will be described with reference to FIGS. FIG. 5 is a block diagram showing the configuration of the imaging apparatus 1A according to an embodiment to which the image processing apparatus, the image processing method, and the image processing program of the present invention are applied.

  FIG. 6 is an explanatory diagram for generating the B @ R plane and the R @ B plane in the second embodiment, FIG. 7 is an explanatory diagram for sampling in the color generation in the second embodiment, and FIG. It is a flowchart showing the procedure of the image processing method and image processing program in 2nd Embodiment.

  Note that the imaging apparatus 1A in the second embodiment is basically the same in configuration as the imaging apparatus 1 shown in the first embodiment, and therefore, the same components are assigned the same reference numerals and detailed description is given. The following is a description of the characteristic features.

  In the imaging unit 2A according to the present invention, the imaging lens 3 is configured to be slidable in the optical axis direction via the lens driving unit 12, and the lens state such as the focal length of the imaging lens 3 and the subject distance from the imaging lens 3 to the subject is set. It is configured to be variable.

  As shown in FIG. 5, an angular velocity sensor (for example, a gyroscope) 15 that detects a shake of the image pickup apparatus 1 </ b> A and outputs an electrical signal (hereinafter referred to as a shake signal) corresponding to the shake amount of the image pickup unit 2 </ b> A. The image processing apparatus 100A is provided with a blur detection unit 40 that detects the blur amount of the imaging apparatus 1A via the angular velocity sensor 15.

  The image processing apparatus 100A also separates the input image (mosaic image) output from the imaging unit 2A for each of R, Gr, Gb, and B pixels, stores the color plane decomposition unit 21, the focal length, and the subject. A lens state detection unit 37 for detecting the lens state of the imaging lens 3 such as a distance, an aberration coefficient storage table 38 storing a distortion aberration coefficient in association with the lens state of the imaging lens 3, a lens state detection unit 37, and an aberration coefficient recording unit. An aberration coefficient setting unit 39 that sets parameters for distortion aberration based on information from 38, blur correction based on detection values from the blur detection unit 40, and aberration correction based on setting values from the aberration coefficient setting unit 39 are performed. A sampling coordinate setting unit 41 for setting a sampling position in the input image is provided.

  Further, the image processing apparatus 100A replaces the interpolation value of the R pixel surrounding the B pixel with the B pixel value in the B pixel arrangement, and sets the R pixel value @B pixel position (hereinafter referred to as R @ B). R @ B plane generation unit 51 for generating the interpolation, the interpolation value of the B pixel surrounding the R pixel is replaced with the R pixel value at the R pixel position, and the B pixel value @R pixel position (hereinafter referred to as B @ R B @ R plane generation unit 52 for generating interpolation as a pixel value at the sampling position of the mosaic image for each of the R plane, Gr plane, Gb plane, B plane, R @ B plane, and B @ R plane. Sampling unit 53 that generates sampling values for each plane, and R plane, Gr plane, Gb plane, B plane, R @ B plane, and B @ R plane generated by sampling unit 53 And a color generation unit 54 generates the color information having a plurality of color component sampling values combined and for each pixel, such as.

  As shown in FIG. 7, the color plane separation unit 21 extracts an R plane (b) in which only R pixels are extracted from a color mosaic image in the Bayer array in (a), and a Gr plane (c) in which only Gr pixels are extracted. , Gb plane (d) from which only Gb pixels are extracted, and B plane (e) from which only B pixels are extracted. At this time, the G pixel arranged in the horizontal direction with respect to the R pixel is designated as Gr, and the G pixel arranged in the horizontal direction with respect to the B pixel is designated as Gb. Further, here, as cited in the first embodiment, the color mosaic image has a luminance of (R, G, B) = (0, 0, 1) in the upper right region via the image boundary L. Suppose that blue and red images are defined and formed so as to have a luminance of (R, G, B) = (1, 0, 0) in the lower left region.

Next, as shown in FIGS. 6A to 6C, the B @ R plane generating unit 51 generates the value of the B pixel at the position of the R pixel in the color mosaic image of FIG. Specifically, the B @ R plane is a luminance signal obtained by re-sampling the B plane at the R pixel position, and is generated by applying the filter of (Equation 16) to the B plane.

For _example, the luminance signal of the B @ R plane at the pixel position of _{A 32} B _uses the pixel values of _{A 21, A 41, A 23} , A 43 around the pixel of the _{A 32, (A 21 × 1} + A 41 × 1 + A ₂₃ × 1 + A ₄₃ × 1) / 4, and the pixel values of A ₂₁ , A ₄₁ , and A ₄₃ are set to 1, and the pixel of A ₂₃ is calculated with reference to FIG. 6B. Substituting the value as 0 gives the luminance signal 3/4.

  As a result, as shown in FIG. 6C, the B @ R plane has a value at the R pixel position, and a value of 0 at other pixel positions excluding the R pixel position.

  Next, as illustrated in FIGS. 6A, 6D, and 6E, the R @ B plane generation unit 52 calculates the value of the R pixel at the position of the B pixel in the color mosaic image in FIG. Generate. Specifically, the R @ B plane is a luminance signal obtained by re-sampling the R plane at the position of the B pixel, and is generated by applying the filter of (Expression 16) to the R plane.

For _example, the luminance signal of R of R @ B _plane, using the pixel values of _{A 32, A 52, A 34} , A 54 around the pixel of the _{A 43} at the pixel position of _{A 43, (A 32 × 1} + A 52 × 1 + A ₃₄ × 1 + A ₅₄ × 1) / 4, and the pixel values of A ₃₂ , A ₅₂ , and A ₅₄ are set to 0 and pixels A ₃₄ with reference to FIG. Substituting the value as 1, the luminance signal 1/4 is obtained.

  As a result, as shown in FIG. 6E, the R @ B plane has a value at the B pixel position, and a value of 0 at other pixel positions excluding the B pixel.

  Next, the sampling coordinate setting unit 41 considers image deformation such as distortion correction, camera shake correction, and digital zoom, and outputs the pixel of the color image to be output based on the parameters acquired by the aberration coefficient setting unit 39 and the shake detection unit 40. Set sampling coordinates on the color mosaic image corresponding to the position.

Here, it is assumed that the pixel position represented by (Expression 17) in the color mosaic image is transformed into the pixel position on the output image represented by (Expression 18). In (Equation 18), (u _s , v _s ) represents the coordinates of the pixel position of the output image, and f represents a deformation function for image deformation.

For example, the pixel position of the color image to be output is (Equation 19) if the zoom magnification is Z times (Equation 19), and is the image deformation of the rotation angle θ (Equation 20). At this time, the sampling coordinate setting unit 41 calculates the sampling coordinates (formula 22) on the mosaic image before the transformation for the output pixel represented by (formula 21). The pixel position is set on an integer grid in the (u, v) coordinate system, but the value of the sampling coordinate calculated in (Equation 22) is not limited to an integer but a real number.

Further, when the chromatic aberration correction is performed in the sampling coordinate setting unit 41, the deformation function f becomes a function f _R , f _G , f _B that is different for each color of _R , _G , _B, and the sampling coordinates are R, G, For each B, (Expression 23), (Expression 24), and (Expression 25). If f is the identity transformation represented by (Equation 26), the same result as in the first embodiment can be obtained.

  Next, as shown in FIG. 7, the sampling unit 53 includes four color planes (R plane, Gr plane, Gb plane, and B plane) generated by the color plane decomposition unit 21 and an R @ B generation unit. Based on the generated R @ B plane and the B @ R plane generated by the B @ R generation unit 51, the sampling value at the sampling position set by the sampling coordinate setting unit 41 (the pixel value for each color at the sampling position) Is calculated).

In other words, as the sampling coordinates, from R plane _{(u sR,} v _sR), from Gr plane _{(u sG,} v _sG), from Gb plane _{(u sG, v sG),} from the B plane _{(u sB,} v _sB), R @ from the B plane _{(u sR, v sR),} B @ from R plane _{(u sB, v sB)} is calculated.

At this time, since the sampling coordinates (u _s , v _s ) are not limited to integers (that is, (u _s , v _s ) does not necessarily match one pixel center), the sampling value is (u _s , v _s ) is calculated by linear interpolation from four valued pixels (pixel values of the same color that each color plane originally has). This interpolation is preferably performed by bilinear interpolation.

  As shown in FIG. 7H, since each color plane has four valued pixels in the form of vertical and horizontal grid points, the four valued pixels surrounding the sampling coordinates 401 to 406 have the sampling coordinates of (100. 8, 101.4), it is located at the apex of a square with a side length of 2 surrounding the sampling coordinates.

That is, for example, if the sampling coordinates (u _sR , v _sR ) for the R plane are (100.8, 101.4), the coordinates (u, v) of the four valued pixels surrounding this are (100, 100). ), (100, 102), (102, 100), (102, 102).

  Therefore, the ratio of the distances between the significant pixels facing each other via the sampling coordinates (here, the u direction is 0.4: 0.6 and the v direction is 0.7: 0.3) is obtained. Using the pixel value of the value pixel, the R pixel value at the sampling position (100.8, 101.4) is calculated by interpolation.

  The pixel values of the four valued pixels are represented by R (100, 100), R (100, 102), R (102, 100), R (102, 102), and sampling positions (100.8, 101. When the R pixel value of 4) is represented by R (100.8, 101.4), the pixel value R (100.8, 101.4) of the sampling coordinate 401 on the R plane is represented by R (100.8). , 101.4) = 0.6 * 0.3 * R (100,100) + 0.6 * 0.7 * R (100,102) + 0.4 * 0.3 * R (102,100) +0. 4 * 0.7 * R (102, 102).

  Gr pixel value at sampling coordinate 402 on Gr plane, Gb pixel value at sampling coordinate 403 on Gb plane, B pixel value at sampling coordinate 404 on B plane, Sampling on R @ B plane The R pixel value at the coordinate 405, the B pixel value at the sampling coordinate on the B @ R plane, and the like can be calculated by interpolation from the pixel values of the four valued pixels surrounding the sampling coordinate as in the R plane.

  The sampling coordinate setting unit 41 sets sampling coordinates in association with all the pixels in the output color image, and the sampling unit 53 calculates sampling values at these all sampling coordinates. Then, this sampling value can be regarded as an image signal by arranging it at the corresponding pixel position of the output color image.

  Next, the color generation unit 54 includes a luminance generation unit 54A that generates luminance at each pixel of the output color image and a color difference generation unit 54B that generates color differences. Note that the function of the luminance generating means according to claim 1 of the present invention is expressed by the luminance generating unit 54A.

  The luminance generation unit 54A calculates the pixel values of Gr, Gb, R, R @ B, B, and B @ R calculated by the sampling unit 53 for each pixel of the output color image 1: 1: (1-k). The luminance signal Y is assumed to be Y = Gr + Gb + (1-k) R + kR @ B + (1-k) B + kB @ R so as to be combined at a ratio of: k: (1-k): k. At this time, the composition ratio of the luminance signal is R: G: B = 1: 2: 1.

  As for the coefficient k used for obtaining the luminance Y, k = 2/3 is optimal for suppressing zipper artifacts, but if k = 2/3, the effect of suppressing zipper artifacts can be obtained. Since the numerical value 2/3 is expressed as 0.101010 ... b in binary number, k is in the range of 1/2 (0.1b in binary number) to 3/4 (0.11b in binary number). In this case, the binary number may be set to a simple numerical value.

Further, the sum R _tot of R and R @ B at luminance Y is expressed by an arithmetic expression of R _tot = (1-k) R + kR @ B, and is transformed into the following expression by incorporating the origin of R @ B. it can.

That is, the sum R _tot (u _sR , v _sR ) of R and R @ B is R _tot (u _sR , v _sR ) = (1−k) * R (u _sR , v _sR ) + k * R @ B ( u _sR , v _sR ) = (1−k) * R (u _sR , v _sR ) + (k / 4) * R (u _sR −1, v _sR −1) + (k / 4) * R (u _sR− 1, _vsR + 1) + (k / 4) * R ( _usR− 1, _vssR + 1) + (k / 4) * R ( _usR− 1, _vsR− 1).

For example, as shown in FIG. 7, taking the value at (u _sR , v _sR ) = (100.8, 101.4) as an example, R _tot (100.8, 101.4) = (1−k ) (0.6 * 0.3 * R (100,100) + 0.6 * 0.7 * R (100,102) + 0.4 * 0.3 * R (102,100) + 0.4 * 0. 7 * R (102,102) + (k / 4) * (0.1 * 0.8 * R (98,100) + 0.1 * 0.2 * R (98,102) + 0.9 * 0. 8 * R (100,100) + 0.9 * 0.2 * R (100,102)) + (k / 4) * (0.1 * 0.8 * R (98,102) + 0.1 * 0 .2 * R (98,104) + 0.9 * 0.8 * R (100,102) + 0.9 * 0.2 * R (100,104)) + (k / 4) * (0.1 * 0.8 * R (100, 00) + 0.1 * 0.2 * R (100,102) + 0.9 * 0.8 * R (102,100) + 0.9 * 0.2 * R (102,102)) + (k / 4 ) * (0.1 * 0.8 * R (100,102) + 0.1 * 0.2 * R (100,104) + 0.9 * 0.8 * R (102,102) + 0.9 * 0 .2 * R (102,104)) = 0.1 * 0.8 * (k / 4) * R (98,100) + 0.1 * (k / 4) * R (98,102) +0.1 * 0.2 * (k / 4) * R (98,104) + (0.8 * (k / 4) + 0.6 * 0.3 * (1-k)) * R (100,100) + ((K / 4) + 0.6 * 0.7 * (1-k)) * R (100,102) + 0.2 * (k / 4) * R (100,104) + (0.9 * 0 .8 * (k / 4) + 0.4 * 0.3 * (1-k ) * R (102,100) + (0.9 * (k / 4) + 0.4 * 0.7 * (1-k)) * R (102,102) + 0.9 * 0.2 * (k / 4) * R (102, 104).

As a result, even if R @ B is not clearly indicated in the arithmetic expression, the value of R in the 5 × 5 pixels around (u _sR , v _sR ) = (100.8, 101.4) It can be seen that the sampling value can be calculated. However, when the R sampling coordinates (u _sR , v _sR ) are exceptionally at the same horizontal and vertical positions as the B pixel, the sampling value is calculated from six or four R pixel values.

  Next, the color difference generation unit 54B generates two color difference signals C1 and C2 from the R, Gr, Gb, and B pixel values (sampling values) generated by the sampling unit 53. Here, C1 is a value obtained by subtracting the R pixel signal and the B pixel signal from the sum of the Gr pixel signal and the Gb pixel signal, while C2 is the R pixel signal and the B pixel. Represents the difference in signal.

  Further, when C1 is generated, the pixel values of R, Gr, Gb, and B are set to (−1 so that the composition ratio of R: G: B = (− 1): 2: (− 1) is obtained. ): 1: 1: (-1), and C1 = (− R + Gr + Gb−B) / 4 is calculated. On the other hand, when C2 is generated, the pixel values of R, Gr, Gb, and B are set to 2: 0: 0 :( so that the composition ratio is R: G: B = 2: 0: (− 2). -2) to calculate C2 = (2R-2B) / 4.

  Further, the color difference generation unit 54B performs correction for suppressing false colors on the color difference generated as described above. As an example of false color suppression, first, a high frequency component K in the vicinity of fs / 2 (Nyquist frequency) is calculated by an arithmetic expression of K = (2Gr−2Gb) / 4. Next, by subtracting the absolute value of K2 from the absolute value of C2 using the following formula, it is possible to suppress red and blue false colors generated in the high-frequency part. C2 → sign (C2) max (0, abs (C2) −abs (K))

  Next, the visual correction unit 34 performs predetermined processing on the luminance signal Y obtained by the luminance signal generation unit 25 and the color difference signals C1 and C2 obtained by the color difference generation unit, and a color composed of these color signals. Correct the appearance of the image. For example, low-frequency filter processing is performed on the color difference signals C1 and C2 to reduce color noise, or high-frequency enhancement is performed on the luminance signal Y to enhance resolution.

  Next, the procedure of the image processing method and the image processing program of the second embodiment will be described with reference to FIG. This procedure is executed by the CPU 18 giving a command signal to each functional unit based on a program stored in the ROM 19. Further, S in FIG. 8 represents a step.

  First, this procedure starts when an activation signal is input to the imaging apparatus 1A or the image processing apparatus 100A by the operator, and then the process proceeds to S101 and S102.

  Next, in S101, an image signal (color mosaic image) is read into the image processing apparatus 100A via the imaging unit 2A, and then the process proceeds to S201.

  In step S201, the color plane separation unit 21 is used to store the R pixel signal, the Gr pixel signal, the Gb pixel signal, and the B pixel signal in association with the Bayer array, and then the process proceeds to S301.

  Next, in S301, the R @ B generation unit 51 and the B @ R generation unit 52 are used to generate the value of the R pixel at the position of the B pixel of the color mosaic image (generate an R @ B plane), and The value of the B pixel at the position of the R pixel of the mosaic image is generated (B @ R plane is generated), and then the process proceeds to S401.

  On the other hand, in step S102, the sampling coordinate setting unit 41 is used to consider image deformation such as distortion correction, camera shake correction, and digital zoom, and output based on the parameters acquired by the aberration coefficient setting unit 39 and the shake detection unit 40. Sampling coordinates on the color mosaic image corresponding to the pixel position of the color image are set, and then the process proceeds to S401.

  Next, in S401, using the sampling unit 53, the four color planes (R plane, Gr plane, Gb plane, and B plane) generated by the color plane separation unit 21 and the R @ B generation unit 51 are generated. Based on the R @ B plane and the B @ R plane generated by the B @ R generation unit 52, the sampling value at the sampling position set by the sampling coordinate setting unit 41 (the pixel value for each color at the sampling position) After that, the process proceeds to S501 and S502.

  Next, in S501, using the luminance generation unit 54A, the pixel values of Gr, Gb, R, R @ B, B, and B @ R calculated by the sampling unit 53 are 1: 1: (1-k): Luminance Y is calculated by the following equation: Y = Gr + Gb + (1-k) R + kR @ B + (1-k) B + kB @ R, so that k: (1-k): k is combined. , The process proceeds to S601.

  On the other hand, in S502, the color difference generation unit 54B is used to generate two color difference signals C1 and C2 from the R, Gr, Gb, and B pixel values (sampling values) generated by the sampling unit 53, and then The process proceeds to S601.

  Next, in S601, it is determined whether or not there is a next sampling coordinate for generating the luminance Y and the color differences C1 and C2, and when there is a next target point (Yes), S401 to 601 are repeated, and the next sampling is performed. When there is no coordinate (No), the process proceeds to S701.

  In step S701, the visual correction unit 34 is used to improve the appearance of the image with respect to the color image signals (luminance Y, color difference C1, and color difference C2) generated by the luminance generation unit 54A and the color difference generation unit 54B. Image correction, conversion to YUV chromaticity coordinates, which are color space coordinates of a standard color image, and the like are performed, and then the process proceeds to S801.

  In step S801, the compression unit 35 is used to compress the digital image signal of the color image output via the visual correction unit 34 by a method such as JPEG (Joint Photographic Experts Group), and the size of the image data at the time of recording. After that, the process proceeds to S901.

  In step S901, the compressed digital image signal is stored in a recording medium such as a flash memory using the recording unit 36, and then the image processing program ends.

  As described above, according to the image processing apparatus 100A, the image processing method, and the image processing program described in the second embodiment, R, Gr, Gb, B from the color mosaic image via the color plane separation unit 21. Then, through the R @ B plane generation unit 51, the interpolation value of the R pixel surrounding the B pixel at the B pixel position is replaced with the B pixel value, and the R pixel value @B Interpolation is generated as the pixel position, and the B pixel value @R pixel is replaced by the R pixel value by replacing the interpolation value of the B pixel surrounding the R pixel at the R pixel position via the B @ R plane generation unit 52. Interpolated as a position, and then, through the sampling unit 53, at the sampling position of the mosaic image associated with the pixel position of the color image, the R plane, Gr plane, Gb plane, B plane, R pixel value Interpolate and generate pixel values for each of the B pixel position plane and B pixel value @ R pixel position plane to generate sampling values for each plane, and then synthesize the sampling values for each color plane via the luminance generation unit 54A Since the luminance of each pixel of the color image is generated, zipper artifacts appearing at the boundary between the red-based image and the blue-based image can be effectively suppressed, and furthermore, the pixel position of the color mosaic image can be suppressed. Regardless, high-quality luminance can be generated at any pixel position.

  In addition, according to the image processing apparatus 100A, the image processing method, and the image processing program described in the second embodiment, based on the blur correction based on the detection value from the blur detection unit 40 and the set value from the aberration coefficient setting unit 39. Since the sampling coordinate setting unit 41 for setting the sampling position in the input image is provided so as to correct the aberration, it is not necessary to correspond each pixel of the color mosaic image and the output color image to 1: 1, and resolution conversion is performed. And various image deformations can be applied at the same time to improve convenience. That is, according to the image processing using the luminance signal generation method of the present embodiment, it is possible to obtain a high-quality luminance signal by suppressing the occurrence of zipper artifacts and jaggies, and to perform desired image deformation. High-quality color images can be generated.

  As mentioned above, although one Example of this invention was described, this invention is not limited to the said Example, It can take a various aspect.

DESCRIPTION OF SYMBOLS 1,1A ... Imaging device, 2, 2A ... Imaging unit, 3 ... Imaging lens, 5 ... Imaging element, 5a ... Bayer arrangement color filter, 6 ... AFE (Analog Front End), 7 ... Correlated double sampling circuit, 8 ... variable gain amplifier (AGC), 9 ... A / D converter, 10 ... detection unit, 11 ... sensor, 12 ... lens driving unit, 13 ... TG (Timing Generator), 18 ... CPU (Central Processing Unit) ), 19... ROM (Read Only Memory), 21... Color plane separation unit, 22... R field memory, 23 a... Gr field memory, 23 b... Gb field memory, 24. SwitchY generator 27 RB plane generation unit, 28 ... luminance correction value generation unit, 29 ... luminance correction unit, 30 ... color difference generation unit, 31 ... C1 generation unit, 32 ... C2 generation unit, 34 ... visual correction unit, 35 ... compression unit, 36 ... Recording unit 37 ... Range state detection unit 38 ... Aberration coefficient storage table 39 ... Aberration coefficient setting unit 40 ... Blur detection unit 41 ... Sampling coordinate setting unit 51 ... R @ B generation unit 52 ... B @ R Generation unit 53... Sampling unit 54. Color generation unit 54 A. Luminance generation unit 54 B Color difference generation unit 100, 100 A Image processing apparatus.

Claims (9)

From a color mosaic image that is output through an image sensor having a color filter with a Bayer array of three colors of R (red), G (green), and B (blue), and each pixel has color information of a single color, pixel by pixel An image processing apparatus for generating a color image having a plurality of color information,
In the Bayer array, when the G arranged next to the R in the horizontal direction is represented as Gr, and the G arranged next to the B is represented as Gb,
A color plane separation unit that separates the color mosaic image into R, Gr, Gb, and B pixels;
An R @ B plane generation unit for generating an interpolation as an R pixel value @ B pixel position by replacing the interpolation value of the R pixel surrounding the B pixel with the B pixel value at the B pixel position;
A B @ R plane generating unit for generating an interpolation as a B pixel value @ R pixel position by replacing the interpolation value of the B pixel surrounding the R pixel with the R pixel value at the R pixel position;
Interpolate pixel values for each of the R plane, Gr plane, Gb plane, B plane, R @ B plane, and B @ R plane at the sampling position of the mosaic image associated with the pixel position of the color image Sampling means for generating and generating sampling values for each plane;
The sampling values of the R plane, Gr plane, Gb plane, B plane, R @ B plane, and B @ R plane generated through the sampling means are combined to generate the luminance of each pixel of the color image. Luminance generation means for
An image processing apparatus comprising: The luminance generating means;
The sampling value of the R plane and the sampling value of the R @ B plane are combined at a ratio of approximately 1: 2.
The image processing apparatus according to claim 1. The luminance generating means;
The sampling value of the B plane and the sampling value of the B @ R plane are combined at a ratio of approximately 1: 2.
The image processing apparatus according to claim 1. From a color mosaic image that is output through an image sensor having a color filter with a Bayer array of three colors of R (red), G (green), and B (blue), and each pixel has color information of a single color, pixel by pixel An image processing method for generating a color image having a plurality of color information,
In the Bayer array, when the G arranged next to the R in the horizontal direction is represented as Gr, and the G arranged next to the B is represented as Gb,
A color plane separation step of separating the color mosaic image for each of R, Gr, Gb, and B pixels;
An R @ B plane generation step of interpolating and generating an R pixel value @ B pixel position by replacing the interpolation value of the R pixel surrounding the B pixel with the B pixel value at the B pixel position;
A B @ R plane generating step of interpolating and generating a B pixel value @ R pixel position by replacing the interpolation value of the B pixel surrounding the R pixel with the R pixel value at the R pixel position;
At the sampling position of the mosaic image associated with the pixel position of the color image, a pixel value is set for each of the R plane, Gr plane, Gb plane, B plane, R @ B plane, and B @ R plane. A sampling step for generating an interpolation and generating a sampling value for each plane;
The sampling values of the R plane, Gr plane, Gb plane, B plane, R @ B plane, and B @ R plane generated through the sampling step are combined to generate the brightness of each pixel of the color image. A luminance generation step to
An image processing method using the method. In the luminance generation step,
The sampling value of the R plane and the sampling value of the R @ B plane are combined at a ratio of approximately 1: 2.
The image processing method according to claim 4. In the luminance generation step,
The sampling value of the B plane and the sampling value of the B @ R plane are combined at a ratio of approximately 1: 2.
The image processing method according to claim 4. From a color mosaic image that is output through an image sensor having a color filter with a Bayer array of three colors of R (red), G (green), and B (blue), and each pixel has color information of a single color, pixel by pixel An image processing program for generating a color image having a plurality of color information,
In the Bayer array, when the G arranged next to the R in the horizontal direction is represented as Gr, and the G arranged next to the B is represented as Gb,
A color plane separation step of separating the color mosaic image for each of R, Gr, Gb, and B pixels;
An R @ B plane generation step of interpolating and generating an R pixel value @ B pixel position by replacing the interpolation value of the R pixel surrounding the B pixel with the B pixel value at the B pixel position;
A B @ R plane generation step of interpolating and generating a B pixel value @ R pixel position by replacing the interpolation value of the B pixel surrounding the R pixel with the R pixel value at the R pixel position;
At the sampling position of the mosaic image associated with the pixel position of the color image, a pixel value is set for each of the R plane, Gr plane, Gb plane, B plane, R @ B plane, and B @ R plane. A sampling step for generating an interpolation and generating a sampling value for each plane;
The sampling values of the R plane, Gr plane, Gb plane, B plane, R @ B plane, and B @ R plane generated through the sampling step are combined to generate the brightness of each pixel of the color image. A luminance generation step to
An image processing program for causing a computer to execute. In the luminance generation step,
The sampling value of the R plane and the sampling value of the R @ B plane are combined at a ratio of approximately 1: 2.
The image processing program according to claim 7, which is executed by a computer as described above. In the luminance generation step,
The sampling value of the B plane and the sampling value of the B @ R plane are combined at a ratio of approximately 1: 2.
The image processing program according to claim 7, which is executed by a computer as described above.

Images